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PREFACE 


Research  to  determine  the  usability  of  DVI  technology  in  computer-based  instruction 
forms  a  portion  of  a  larger  research  and  development  (R&D)  program  being  conducted  by  the 
Instructional  Design  Branch  (AL/HRTC)  of  the  Technical  Training  Research  Division,  Human 
Resources  Directorate  of  the  Armstrong  Laboratory.  R&D  by  this  branch  has  investigated  the 
potential  for  providing  guidelines,  authoring  assistance  and  automated  tools  to  Air  Force 
instructional  developers.  This  project  provides  some  lessons  learned  in  the  application  of  a 
rapidly  evolving  technology  to  typical  Air  Force  instructional  design  problems.  The  results  are 
already  being  applied  in  continuing  research  applications. 

When  this  project  began,  multimedia  technology  in  general  (both  software  and  hardware), 
and  DVI  specifically,  was  changing  rapidly.  Software  tools  were  being  developed  by  several 
vendors.  The  research  team  knew  that  even  though  some  products  were  not  complete  the  project 
had  to  begin.  As  a  consequence,  several  decisions  were  made  early  in  the  project  which  directed 
its  course.  These  decisions  centered  around  a  probable  typical  Air  Force  multimedia  delivery 
platform  of  the  future  being  a  3/486  processor,  relatively  large  hard  disk  (>500MB),  optional 
CD-ROM,  Windows-based  machine.  These  assumptions  influenced  the  choices  made 
throughout  the  study.  Within  6  months  of  beginning  this  project  the  researchers  realized  that  the 
initial  goal  of  the  study,  i.e.,  to  determine  the  applicability  of  DVI  technology  to  computer-based 
training  was  too  limited.  Developments  were  taking  place  in  the  marketplace  which  permitted 
software  driven  digital  video  capture,  editing  and  replay  in  the  Windows  environment. 

Therefore,  the  research  team  expanded  the  research  effort  to  include  looking  at  this  technology. 
Results  are  included  in  this  report. 


Special  thanks  from  the  entire  research  team  must  be  extended  to  Lt.Col.  James  Mohan  of 
the  338th  Operations  Training  Squadron,  Air  Training  Command,  who  served  as  the  principal 
subject  matter  expert  (SME)  for  this  project.  Lt.  Col.  Mohan  not  only  provided  the  required 
formation  flying  expertise  necessary  to  produce  accurate  lessons;  at  times  he  was  part  of  the 
research  team,  discussing  problems  associated  with  hardware  or  software,  offering  instructional 
systems  design  recommendations,  and  providing  insights  into  what  would  and  would  not  work, 
thereby  saving  the  development  team  many  long  hours  of  trial  and  error.  He  was  a  welcome  and 
active  participant  at  all  meetings,  providing  helpful  comments  to  the  development  team 
members,  yet  maintaining  his  role  as  the  principal  SME. 


We  would  also  like  to  thank  Capt.  John  O'Connell  of  the  338th  Operations  Training 
Squadron,  who  provided  additional  expertise  in  formation  flying.  Video  editing  equipment  and 

services  were  provided  by  Alejandro  Maya  of  VideoWave  Productions. . - 7 — 

(  Acce:  Kj;  ~ 

[  NTlS 


l 

t 

i 


duo 

Ur.air..v.  • 


C 


f 

j  Ry 

j  Di  t  , 
f 

I  Aw  .. 

i  ! 


SUMMARY 


Multimedia  technology  and  Digital  Video  Interactive  (DVI)  in  particular  are  maturing 
visualization  technologies  which  hold  promise  as  powerful  tools  for  Air  Force  instructional 
developers.  The  difficulty  with  multimedia  is  that  it  requires  expertise  in  several  specialties  to 
work  with  it  effectively.  Few  Air  Force  instructional  developers  or  instructional  development 
shops  possess  the  skills  needed  to  integrate  multimedia  into  training  applications.  This  research 
project  examined  the  roles  of  a  multimedia  development  team  to  determine  the  required 
functions,  activities  and  tasks  performed,  and  the  skills  and  knowledge  which  support 
performance  of  such  tasks. 

As  part  of  the  research,  several  prototype  multimedia  interactive  courseware  lessons  were 
developed  to  demonstrate  multimedia  development  and  production  activities.  The  original  focus 
of  the  project  was  to  determine  the  usability  of  DVI  technology  in  computer-based  training, 
however,  the  development  team  also  examined  other  software  packages  which  performed  similar 
functions.  Thus  the  focus  was  broadened  to  include  multimedia  rather  than  only  DVI. 

An  observational  study  was  conducted  during  the  entire  multimedia  development  project. 
During  the  observations,  critical  incidents  which  had  an  impact  on  the  project  were  identified. 
Results  of  the  observations  were  also  used  to  debrief  the  multimedia  expert  and  team  members. 

A  task  analysis  hierarchy  was  constructed  describing  multimedia  development  activities.  The 
observations  and  task  hierarchy  formed  the  basis  for  development  of  a  model  of  the  multimedia 
development  process.  While  the  model  can  stand  alone  as  a  description  of  multimedia 
development,  it  should  be  considered  the  first  step  at  formulating  principles  and  guidelines  for 
multimedia  development.  The  model  also  serves  a  second  purpose  as  the  basis  for  identifying 
components  of  the  multimedia  process  which  can  be  automated.  In  the  future,  inclusion  of 
visualization  technologies  such  as  multimedia  into  automated  systems  such  as  the  Automated 
Instructional  Design  Advisor  (AIDA)  will  encourage  the  effective  use  of  such  powerful 
technologies  to  support  sound  instructional  design. 
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DETERMINING  THE  USABILITY  OF 
DIGITAL  VIDEO  INTERACTIVE  (DVI)  TECHNOLOGY 
IN  COMPUTER-BASED  INSTRUCTION 


I.  INTRODUCTION 

Armstrong  Laboratory  has  been  conducting  research  into  the  simplification  of  the 
instructional  design  process  for  several  years  (Kintsch,  E.,  Tennyson,  R.  D.,  Gagn'e,  R.,  & 
Muraida,  D.  J.  (1991);  Poison,  M.  E.,  Poison,  P.  G.,  Kieras,  D.,  Jalff,  J.,  &  Hioki,  R.  (1992), 
Tennyson,  R.  D.,  Reigeluth,  C.  R.,  &  Gettman,  D.  J.,  (1992).  Computer-Based  Training  (CBT) 
including  some  form  of  multimedia  is  a  powerful  tool  when  designed  and  implemented  by  a 
trained  instructional  developer.  However,  the  availability  of  skilled  instructional  developers  in 
the  Air  Force  is  limited.  Traditionally,  the  Air  Force  relies  on  a  few  instructional  developers  to 
assist  many  subject  matter  experts  (SMEs)  to  develop  all  kinds  of  training.  Although  fully 
qualified  in  their  areas  of  expertise,  SMEs  rarely  possess  both  adequate  instructional  design 
knowledge  and  the  various  computer  skills  required  to  develop  an  effective  training  system.  If 
tools  were  available  to  assist  the  instructional  designer,  i.e.  clone  the  expert  instructional 
designer,  more  high  quality  CBT  could  be  produced  faster  and  cheaper  than  with  current 
practices.  The  Laboratory’s  work  on  projects  such  as  the  Automated  Instructional  Design 
Advisor  (AIDA)  is  on  the  verge  of  providing  a  breakthrough  in  the  area  of  such  instructional 
design  tools.  What  is  also  needed  is  the  ability  to  integrate  the  highly  creative  capability  of 
multimedia  into  systems  such  as  AIDA.  This  will  require  identifying  and  capturing  the 
instructional  design  and  technical  expertise  associated  with  multimedia  and  ensuring  that  this 
expertise  is  also  built  into  AIDA. 

This  study  used  direct  observation  methods,  job  and  task  analysis,  and  structured 
interview  techniques  to  examine  the  process  of  developing  a  multimedia  prototype  training 
system.  The  purpose  of  using  observational  methods  was  to  formulate  a  conceptual  model  of  the 
multimedia  development  process  to  serve  as  a  basis  for  integrating  research  findings  with 
computer-based  courseware  authoring  aids  such  as  AIDA.  Hypotheses  were  proposed 
concerning  the  critical  information  about  multimedia  development  required  by  Air  Force  SMEs. 
The  study  also  identified  critical  activities  and  development  team  interactions  required  to 
produce  cost-efficient  and  effective  training  courseware. 

Background 

Multimedia  platforms,  including  interactive  videodisc,  digital  video,  and  specific 
products  such  as  Digital  Video  Interactive  (DVI)  are  becoming  more  widely  used.  The  goal  of 
these  multimedia  delivery  systems  is  to  take  advantage  of  the  integrated  interface  and  interactive 
opportunities  which  involve  the  student  in  an  environment  which  is  as  close  as  possible  to  the 
context  in  which  learning  will  be  applied.  However,  exceptionally  well  analyzed  and 
professionally  prepared  implementation  of  multimedia  technologies  are  not  often  replicated  in 
typical  instructional  settings.  Two  possible  reasons  for  this  problem  have  to  do  with  the 
availability  of  well-documented  and  compatible  software  products,  and  the  relative  difficulty  of 
learning  to  effectively  use  multimedia  tools  to  enhance  sound  instructional  design. 
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In  keeping  with  Armstrong  Laboratory's  long  history  of  developing  automated 
instructional  design  tools,  AIDA  will  provide  typical  Air  Force  SMEs,  i.e.,  inexperienced 
instructional  designers,  with  the  necessary  tools  to  assist  them  in  producing  consistently  high 
quality  and  less  costly  courseware  by  providing  intelligent  frameworks  appropriate  for  a  variety 
of  training  settings.  The  AIDA  tool  kit  can  be  enhanced  by  including  visualization  technologies 
such  as  multimedia,  which  has  the  ability  to  produce  highly  realistic  job-like  simulations.  The 
primary  objective  of  the  current  study  was  to  investigate  the  multimedia  development  process, 
specifically  DVI,  in  order  to  extract  general  procedures,  guidelines,  and  techniques  which  could 
be  used  by:  1)  Air  Force  instructional  developers/SMEs  required  to  perform  multimedia  tasks, 
and;  2)  software  designers  of  computer-based  courseware  authoring  aids  such  as  AIDA.  This 
study  examined  the  multimedia  development  process  in  detail  to  identify  those  activities,  tasks, 
tools,  and  other  elements  such  as  hardware  and  software  selection,  specific  to  the  training 
medium  which  can  be  modeled.  Applications  of  these  results  will  provide  a  framework  for 
demonstrating  to  novice  instructional  developers  the  proper  developmental  context  and 
utilization  of  the  medium,  and  identify  for  software  developers  unique  multimedia  characteristics 
which  can  be  successfully  integrated  into  tools  such  as  AIDA. 

Approach 

This  study  began  with  a  comprehensive  review  of  the  literature  in  three  primary  areas:  1) 
observational  methods;  2)  problem-solving  research,  and;  3)  multimedia  development. 
Additionally,  the  status  of  various  emerging  training  technologies  were  reviewed.  The  approach 
was  guided  by  three  general  principles.  First,  since  this  was  an  exploratory  research  project,  it 
emphasized  collecting  basic  data  on  the  rapidly  evolving  topic  of  multimedia  development. 
Second,  certain  observational  methods,  such  as  the  critical  incident  technique,  were  suggested  by 
the  Laboratory  in  part  to  try  them  out  for  future,  more  intensive  studies  of  developmental 
processes.  Finally,  the  cognitive  relationships  of  instructional  design  to  multimedia  development 
were  evaluated  in  terms  of  their  inputs,  processes,  and  outputs  for  modeling  training  system 
development  (Spector,  Muraida,  &  Marlino  1992). 


Observational  research  is  often  conducted  to  obtain  background  data  for  understanding 
behavior  and  performance  or  to  formulate  an  initial  conceptualization  of  the  problem  domain. 
Traditional  direct  observational  methods  are  used  in  psychological,  anthropological,  educational, 
and  consumer  research.  They  usually  involve  minimal  interference  with  the  subject's  behavior 
and  environment.  Traditional  observational  data,  often  described  as  "field  notes,"  usually  consist 
of  two  parts—  description  and  reflection.  Description  includes  details  about  the  subject,  setting, 
accounts  of  particular  events  and  activities,  and  relevant  dialog.  Reflection  can  contain 
preliminary  analysis,  decisions  about  methods,  actual  or  potential  conflicts,  and  points  of 
clarification  (Bogdan  &  Biklen  1992). 


There  are  several  limitations  inherent  in  observational  research.  First,  it  is  basically  a 
subjective  method,  involving  individual  perception  and  judgment.  Second,  the  mere  presence  of 
an  observer  may  have  an  affect  on  subjects'  behaviors.  Third,  a  variety  of  observational  errors 
can  occur,  including:  loss  of  detail  through  simplification,  stereotyping  and  prejudice,  preference 
for  familiar  behaviors,  and  misinterpreting  or  second-guessing  the  thoughts  or  motives  of 
subjects  (Meister  1985).  It  is  also  more  difficult  to  observe,  in  real-time,  a  group  of  individuals 
rather  than  just  one  person.  These  kinds  of  factors  and  errors  can  have  multiple  effects  on  the 
research  data  collected,  although  these  are  typically  hard  to  validate  (Kirk  &  Miller  1986). 

Fourth,  interpretation  of  observations  can  be  problematic.  Finally,  reliability  is  often  an  issue  in 
observational  studies,  however,  since  the  current  research  study  was  a  pilot  project  and  there  was 
only  one  observer,  no  tests  of  reliability  were  conducted. 

The  structured  or  informational  interview  is  a  technique  which  was  also  used  to  obtain 
information  during  the  multimedia  development  process.  According  to  Meister  (1985),  it  can 
help  to  evaluate  the  subject's:  1)  internal  processes,  e.g.,  what  he  or  she  thinks  he  or  she  is  doing; 
2)  knowledge  of  how  a  job  is  performed,  and;  3)  perception  of  external  events.  Questions  must 
be  clear  and  concise,  although  the  interview  may  be  informally  conducted,  e.g.,  questions  can 
lead  to  unplanned  but  relevant  topics. 

Problem  Solving  Research 

To  facilitate  the  transition  between  old  and  new  training  technologies,  several  basic 
research  projects  have  been  examined  with  regard  to  how  novices  and  experts  approach  problem 
solving,  decision  making,  and  trouble-shooting  mechanized  systems  (Halff,  Hollan  &  Hutchins 
1986).  In  these  cases,  instructional  design  was  considered  a  complex  problem  solving  activity 
directed  toward  an  equivocal  problem  (Duchastel  1990,  Rowland  1992).  The  purpose  of 
modeling  instructional  design  in  this  context  is  to  determine  what  cognitive  sources  of  difficulty 
affect  the  instructional  designer  or  multimedia  developer  when  the  requirements  for  using  an 
interface  to  computer-based  tools  are  superimposed  on  the  demands  of  the  instructional  design 
process.  Studies  have  revealed  that  a  large,  organized  body  of  domain  knowledge  is  a 
prerequisite  to  expertise  (Bedard  &  Chi  1992,  Larkin  1979).  In  general,  experts  tend  to  perform 
quickly,  strategically,  and  automatically,  and  base  the  organization  of  their  knowledge  on 
meaning.  Experts  tend  to  work  in  iterative,  knowledge  building  cycles.  On  the  other  hand, 
novices  work  slowly  and  base  their  organization  of  knowledge  on  the  surface  features  of  the 
information  presented  (Anderson  1985,  Steptich  1991). 

The  complex  problem  solving  process  used  by  experts  involved  in  instructional  design 
must  be  coordinated  with  knowledge  about  specific  subject  matter  and  current  technology.  For 
example,  little  is  known  about  the  cognitive  demands  placed  on  the  multimedia  developer.  If  the 
Air  Force  or  for  that  matter,  any  user  intends  to  successfully  use  multimedia  technologies  for 
instruction,  the  technology  must  be  fully  understood  (Kearsley  1991).  The  primary  objective  of 
this  research  was  to  investigate  the  multimedia  development  process  in  order  to  extract  general 
procedures,  principles,  and  techniques  which  will  assist  novices  to  develop  CBT  either  in  the 
form  of  guidelines  or  when  incorporated  into  automated  tools. 
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One  method  used  to  investigate  multimedia  development  was  cognitive  task  analysis. 
Cognitive  task  analysis  uses  a  combination  of  structured  interviews  and  thinking  aloud  verbal 
reports  to  investigate  both  the  relationships  between  tasks  and  between  people  demonstrating 
various  levels  of  competence  performing  these  tasks.  It  may  reveal  cognitive  skills  that  cannot 
be  directly  observed,  and  of  which  people  may  not  be  aware.  For  example,  instead  of  asking 
SMEs  to  list  the  kinds  of  problems  encountered  on  the  job  and  the  skills  required  to  solve  them, 
they  would  be  asked  to  formally  generate  and  solve  similar  problems  (Bonner  1988,  Means  & 
Gott  1988).  Cognitive  data,  emphasizing  internal,  cognitive  structures  and  processes,  can 
contribute  to  traditional  task  analysis  by  providing  added  levels  of  details  to  derive  learning 
objectives  and  assessment  measures  for  cognitively  based  tasks  such  as  prioritizing,  evaluating, 
and  decision  making  (Schlager,  Means  &  Roth  1990). 

Multimedia  Development  Research 

Multimedia  has  been  used  for  more  than  30  years  in  libraries  and  audio  visual 
departments  of  schools,  government,  and  industry.  However,  since  1987,  with  the  advent  of 
digital  video  technology,  the  term  multimedia  has  been  used  to  mean  the  integration  of  all  media- 
-  text  still  images,  graphics,  audio,  and  full  motion  video.  More  recently,  the  term’s  definition 
has  been  ambiguous;  multimedia  has  been  used  more  as  a  marketing  term  to  cover  a  broad  range 
of  products  which  provide  little  more  than  pictures  with  text.  For  this  project,  multimedia  is 
defined  as  the  integration  of  all  media  in  a  digital  environment  for  interactive  use.  The  process 
of  creating  applications  which  integrate  these  elements  is  the  multimedia  development  process. 

As  DVI  and  other  multimedia  technologies  become  increasingly  supported  by  software 
tools,  more  references  are  available  regarding  equipment,  technological  processes,  technical,  and 
creative  roles  (Burger  1993,  Bunzel  and  Morris  1992,  Luther  1991).  These  three  texts  offer 
comprehensive  presentations  on  DVI  technology,  software,  hardware,  authoring  systems,  and 
special  effects.  Ripley  (1990)  also  effectively  summarizes  the  challenges  of  working  with  digital 
presentation  technologies.  A  technical  paper  by  Gavora  (1991)  focuses  on  how  learning 
behaviors  and  instructional  systems  are  affected  by  digital  multimedia  delivery.  However,  few 
observational  studies  have  documented  the  complex  multimedia  development  process.  One 
study  by  Hribar  et  al.  (1992)  describes  unique  considerations  for  using  DVI  to  develop  a 
submarine  training  system,  including  assembling  a  multimedia  development  team.  Other 
research  issues  defined  were:  1)  importance  of  understanding  the  technology;  2)  still  image 
resolution,  size,  and  format;  3)  original  image  source;  4)  use  of  video,  and;  5)  authoring 
language. 


H.  METHODS 
Observational  Data 

The  observational  component  of  this  study  focused  on  accomplishing  three  goals:  1) 
providing  a  preliminary  description  of  the  competencies  required  of  an  expert  multimedia 
developer;  2)  documenting  multimedia  development  and  production  and;  3)  producing  an 
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empirically  testable  model  of  multimedia  development.  Observational  methods,  such  as  task 
analysis,  and  structured  interviews,  used  to  obtain  a  large  amount  of  qualitative  data,  were 
appropriate  during  this  initial  stage  of  research  to  define  the  kinds  of  knowledge,  skills,  abilities, 
and  relevant  background  experience  used  by  an  expert  multimedia  developer.  In  addition,  these 
same  methods  were  also  effective  at  analyzing  the  working  relationships  between  development 
team  members  (e.g.,  instructional  developers,  programmers  and  SMEs),  as  well  as  the  type, 
organization,  and  sequence  of  activities  required  to  produce  multimedia  courseware. 

A  research  analyst  was  dedicated  to  observing  the  participants  involved  in  multimedia 
development  during  most  scheduled  activities,  ranging  from  preliminary  team  meetings  and  tape 
mastering  sessions,  to  a  final  demonstration  of  the  prototype  lessons.  Informal  activities  and 
interactions  were  also  observed  during  several  technical  discussions  and  programming  sessions. 
Most  sessions  were  audio  taped  and  supplemented  by  extensive  written  notes.  Details  were 
confirmed  or  modified,  if  necessary,  during  regularly  scheduled  meetings  with  the  multimedia 
development  expert  using  direct  questioning.  Problems  were  clarified  and  additional  information 
was  elicited.  Behavioral  samples  were  obtained  to  identify  skills  and  abilities  used  by  team 
members,  how  they  interacted  with  each  other,  and  how,  generally,  behaviors  changed  through 
time.  All  versions  of  paper  documents  produced  during  multimedia  development,  such  as 
storyboards,  milestone  charts,  narration  scripts,  etc.  were  dated  and  catalogued. 

During  the  early  weeks  of  the  research  project,  the  observer  conducted  a  warm-up  session 
for  verbal  reports.  The  purpose  was  to  introduce  everyone  to  the  formal  knowledge  elicitation 
process,  and  to  help  them  feel  comfortable  thinking  aloud  in  both  current  and  retrospective 
modes.  The  observer  picked  two  problems  from  Ericsson  and  Simon's  (1984)  Protocol  Analysis : 
one  involved  verbally  solving  a  complex  multiplication  problem;  the  other  involved  verbalizing 
the  process  of  thinking  about  his  parents'  house  and  counting  the  number  of  windows.  After 
these  sessions  the  observer  gave  feedback  about  the  responses.  Feedback  contained  guidance 
about  the  thinking  aloud  process  rather  than  qualitative  judgments.  This  brief  warm-up  session 
served  to  establish  the  ground  rules  for  future  debriefings. 

All  scheduled  meetings  were  audio  taped.  With  the  exception  of  two  trial  transcriptions 
during  the  early  pan  of  the  process,  these  audio  tapes  were  not  re-played  or  transcribed  due  to  the 
effort  which  would  be  necessary  to  code  and  analyze  the  transcription.  They  were  available, 
however,  in  case  questions  arose  about  events.  Occasionally,  the  multimedia  development  team 
worked  when  no  observer  was  available.  In  these  cases,  the  observer  obtained  data  during 
debriefing  of  participants,  and  also  collected  written  notes  or  other  materials  pertaining  to  the 
status  of  lesson  development. 


Knowledge  Elicitation 

Knowledge  elicitation  methods  are  techniques  for  knowledge  acquisition  dealing 
specifically  with  the  collection  and  analysis  of  material  obtained  during  direct  interaction  with  a 
subject  matter  expert  and  producing  a  model  of  that  knowledge  (Pidgeon,  N.  F.,  Turner,  B.  A.,  & 
Blockley,  D.  I.,  (1991).  A  well-known  example  of  a  formal  knowledge  elicitation  method  and 
cognitive  process  coding  scheme  is  protocol  analysis  using  verbal  data  (Ericsson,  K.  A.,  & 
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Simon,  H.  A.,  (1984).  Verbal  data  are  derived  directly  from  observations  which  then  must  be 
clearly  and  objectively  coded  based  on  previously  defined  criteria.  Data  can  be  recorded  for 
transcription  and  coding.  If  only  general  impressions  are  required,  an  observer  can  complete 
simple  checklists,  or  maintain  a  narrative  record  of  events  and  activities.  However,  transcripts 
are  useful  for  making  the  encoding  process  more  explicit  and  objective  since  they  ensure  that  all, 
rather  than  select  data  are  recorded.  Observers  who  code  behaviors  as  they  occur,  without 
reference  documentation,  may  eventually  make  subjective  judgments  or  end  up  with  an 
incomplete  record. 

Knowledge  elicitation  is  an  extremely  complicated  and  labor  intensive  process  because  it 
is  often  difficult  for  an  expert  to  describe  knowledge  which  has  become  tacit,  intuitive,  and  hard 
to  communicate  about.  Cooke’s  (1992)  taxonomy  and  evaluation  of  knowledge  elicitation 
techniques  is  an  excellent  review  of  the  literature  in  this  area.  She  classifies  '’direct  methods,”  as 
techniques  involving  direct  reporting  of  verbalizable  knowledge,  such  as  interviews, 
questionnaires,  simple  observations,  and  think  aloud  protocols.  Unfortunately,  many  techniques 
lack  face  validity  and  are  unable  to  combine  representations  of  knowledge  from  multiple  experts 
(Cooke  1992). 

Informal  methods  of  observation  were  very  useful  for  the  current  research  project. 
Initially,  they  promoted  rapport  between  the  observer,  the  multimedia  expert  and  other 
development  team  members,  and  helped  to  define  the  domain  of  knowledge  itself  (Ford  &  Wood 
1992).  At  the  beginning  of  the  project,  use  of  these  techniques  did  not  require  an  observer 
conversant  in  multimedia.  However,  both  multimedia  and  instructional  design  expertise  were 
required  in  converting  the  raw  data  into  a  comprehensive  model  of  multimedia  development. 

Critical  Incident  Technique 

The  critical  incident  (Cl)  technique  (Flanagan  1954),  which  distinguishes  between 
effective  and  ineffective  goal  directed  behaviors,  was  used  as  a  framework  to  organize  much  of 
the  data.  The  approach  provided  useful  information  about  reasons  for  problems,  common  errors, 
and  alternative  strategies.  The  technique  utilizes  a  set  of  procedures  for  cc fleeting  and  ranking 
important  facts  (based  on  observations),  and  organizes  them  into  a  series  of  relationships  that  can 
be  tested  by  further  observations  in  more  controlled  studies.  The  Cl  technique  uses  rules  which 
strengthen  objectivity  and  enhance  reliability.  This  includes  basing  research  goals  on  a  general 
aim,  along  with  simple  non-interpretative  judgments  performed  only  by  qualified  observers 
(Flanagan  1954). 

Analysis  of  CIs  can  also  be  supplemented  by  cognitive  process  coding  schemes,  also 
called  protocol  analysis,  verbal  reports,  or  verbal  analysis  (Hamm  1987;  Lusk,  Smith,  and  Neal 
1988)  as  well  as  by  structured  interviews  and  administration  of  questionnaires.  The  goal  of  these 
kinds  of  methods  is  to  evaluate  the  subject's  cognitive  processes  by  verbal  methods  (Ericsson  and 
Simon  1984).  Verbal  data  are  derived  directly  from  observations  which  then  must  be  clearly  and 
objectively  coded  based  on  previously  defined  criteria.  Evaluation  of  cognitive  processes  in  the 
context  of  task  analysis  is  most  appropriate  for  developing  or  testing  models  of  cognition.  While 
this  research  project  initially  planned  to  conduct  protocol  analysis  on  selected  transcripts  on  an 
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experimental  basis,  the  process  of  transcription  and  coding  proved  too  complex  and  time 
consuming  to  justify  for  this  preliminary  research  effort. 

PowmentatUm  otProgsres 


Job  and  Task  Analysis 

Job  analysis  identifies  the  requirements  and  tasks  associated  with  a  particular  job, 
whereas  task  analysis  breaks  down  individual  tasks  into  their  component  parts  or  subtasks  in 
order  to  identify  the  behavioral  implications  of  the  task,  the  functions  of  the  expert,  and  the 
relationship  of  the  task  to  the  job  in  general  (McGraw  &  Harbison-Briggs  1989).  There  are  no 
previous  studies  available  on  job  or  task  analysis  of  multimedia  designers  with  which  to  compare 
the  results  of  the  current  study.  One  goal  of  job  and  task  analysis  was  to  define  the  principal 
roles,  organization,  structure,  individual  components,  and  sequential  relationships  between  the 
specific  duties  of  the  multimedia  developer  and  other  development  team  personnel.  Additional 
information  was  obtained  about  skills  and  abilities,  branching  points,  problem  solving  and 
decision  steps,  communication  between  development  team  members,  perceived  relative 
importance  of  tasks,  and  perceived  need  for  specialized  background  training  (Meister  1985). 

Data  were  also  obtained  about  required  specialized  equipment,  tools,  materials,  and 
environmental  setting.  Data  pertaining  to  multimedia  development  were  categorized  into  five 
major  phases  (roughly  analogous  to  Instructional  Systems  Development  [ISD]):  analysis,  design, 
development/production,  authoring,  and  testing.  In  particular,  production  encompasses  many 
activities  unique  to  multimedia  development  in  general  and  DVI  specifically,  compared  with 
similar  interactive  courseware  development  projects. 

Several  roles  can  be  identified  on  the  multimedia  development  team.  While  each  of  these 
may  require  unique  adaptations  of  individual  tasks,  in  general,  each  role  is  similar  or  identical  to 
those  on  traditional  courseware  development  teams.  The  uniqueness  of  multimedia  comes  from 
the  incorporation  of  video  elements  into  the  courseware  development  process.  In  the  following 
sections  the  various  roles  of  the  multimedia  development  team  are  described.  In  general, 
depending  upon  the  size  and  complexity  of  the  project,  these  roles  can  be  filled  by  a  single 
individual,  several  individuals  or  one  individual  may  fill  more  than  one  role.  In  any  event,  these 
functions  are  ones  that  normally  must  be  performed.  To  the  extent  that  the  multimedia 
development  team  possesses  the  education  and  experience  combination  necessary  to  perform  all 
functions  successfully,  the  team  and  the  interactive  courseware  produced  will  be  successful. 

Project  Manager.  The  project  manager  or  project  administrator  normally  organizes  the 
project  and  the  team  and  may  also  have  day-to-day  project  management  responsibilities.  In  order 
to  properly  perform  project  planning,  the  project  manager  must  be  aware  of  all  aspects  of 
multimedia  development,  how  long  various  processes  take,  and  who  is  qualified  to  work  on  what 
tasks.  Communication  skills  are  essential  for  coordinating  client  needs,  managing 
subcontractors,  negotiating  with  vendors,  controlling  expenses,  and  allowing  for  the  expression 
of  creative  concepts  among  team  members  within  project  constraints.  The  project  manager 
should  monitor  technology  and  project  production  schedules.  Checks  should  be  put  in  place  to 
ensure  the  timely  completion  of  stoxyboards,  shot  sheets,  flowcharts,  and  other  multimedia 
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products  so  that  the  application  meets  the  needs  of  the  client.  The  project  manager  is  ultimately 
responsible  for  the  quality  of  the  interactive  courseware  product,  therefore,  he  should  put  those 
quality  control  measures  into  effect  which  are  most  effective  for  the  team  and  the  project. 

Analyst/Knowledge  Engineer.  The  project  analyst  or  knowledge  engineer’s  job  is  to 
articulate  on  paper  the  content  of  the  application  so  that  other  team  members  can  develop  and 
build  on  the  application.  Content  originates  from  observing  and/or  interviewing  SMEs, 
researching  technical  publications,  manuals,  existing  training  materials,  videos,  or  any  other 
available  sources  which  .accurately  describe  the  job.  It  is  useful  to  have  the  ability  to  extrapolate 
knowledge  from  observations,  have  insight  into  underlying  covert  behaviors,  and  to  organize  real 
world  experiences  into  useful  descriptive  information.  Highly  refined  listening  and  organization 
skills  are  essential. 

Graphic  Artist.  The  role  of  the  graphic  artist  is  to  produce  the  graphic  designs,  templates, 
textures,  colors,  graduated  backgrounds,  typography,  and  transitions  used  in  the  interactive 
courseware.  Relevant  background  should  include  knowledge  of  computer  graphics,  a  full 
appreciation  of  the  capabilities  and  limitations  of  screen  dimensions  and  cathode  ray  tube  (CRT) 
or  backlight  effect  of  colors  on  human  interface.  Interface  design  issues  must  be  understood  in 
order  to  create  screens,  buttons,  or  menus,  as  well  as  a  familiarity  with  aspects  of  visual 
perception  as  they  apply  to  training  systems.  The  graphic  artist  needs  to  know  about  image 
capture,  including  how  to  scale  and  crop  and  images,  how  to  select  one  image  for  use  over 
another,  and  which  parts  of  an  image  to  use.  Multimedia  provides  the  software  tools  for  image 
digitization,  while  various  software  packages  enhance  these  capabilities.  Ability  to  work  with 
and  manage  large  quantities  of  graphic  data  is  useful  since  multimedia  applications  often  use 
numerous  images. 

Instructional  Designer.  An  instructional  designer  traditionally  works  with  the  SME  to 
ensure  that  the  training  system  satisfies  the  requirements  of  the  cliei...  This  job  requires  a 
background  in  ISD  and  educational  technology,  including  knowledge  of  information  delivery  and  , 
cognitive  issues.  The  instructional  designer  should  have  experience  with  user  interface 
techniques  in  order  to  define  such  things  as  lesson  navigation  and  rules,  looping  sequences, 
escape  (pause),  help,  and  more  information  support.  Ideally,  additional  knowledge  should 
include  the  status  of  emerging  technologies,  expert  systems,  intelligent  tutoring  systems, 
hypermedia,  and  multimedia.  The  instructional  designer  should  also  be  familiar  with 
programming,  authoring,  and  the  structured  design  of  software  for  training  applications. 

Programmer.  The  Programmer  should  understand  the  courseware  development  process 
and  other  issues  associated  with  designing,  programming,  and  testing  courseware.  The 
programmer  should  also  know  multimedia  technology  (hardware  and  software),  and  the 
programming  language  to  be  utilized  (in  this  case  C  or  C++);  if  using  an  authoring  environment 
(or  higher  level  programming  language),  proficiency  with  the  authoring  language  is  important. 

The  programmer  should  be  able  to  work  with  the  team  to  interpret  storyboards,  flowcharts,  and 
scripts  which  will  be  used  to  link  the  multimedia  elements.  Programming  or  authoring  a 
multimedia  application  becomes  an  intensely  exciting  activity  when  it  is  time  to  integrate  all  the 
multimedia  elements  into  the  final  product. 
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Technician.  A  computer/hardware  technician  is  an  important  member  of  the  team.  This 
individual  coordinates  acquisition,  installation  and  setup  of  equipment,  upgrades,  system 
modifications,  compatibility  problems,  networking  configuration,  and  maintenance.  The 
computer/hardware  technician  needs  to  have  a  basic  knowledge  of  existing  hardware  and 
software  plus  experience  in  working  with  computers,  peripherals,  and  audio  and  video 
interconnects.  Although  this  appears  to  be  a  relatively  minor  role  at  first  glance,  the 
computer/hardware  technician  is  involved  throughout  the  multimedia  development  process  in 
nearly  all  activities. 

Production  Specialists.  The  complete  multimedia  development  team  includes  a  variety  of 
production  specialty  roles.  Traditional  roles  are:  video  producer,  responsible  for  coordinating 
talent  and  equipment  requirements  for  video  production.  This  individual  should  have  some 
multimedia  and  directing  experience  in  addition  to  the  ability  to  plan,  manage,  and  supervise 
video  production.  The  photographer  integrates  design  and  content  requirements,  and  takes  care 
of  lighting  and  close-up  photography.  Again,  it  cannot  be  emphasized  enough  that  these  roles 
cannot  be  adequately  filled  by  the  typical  man-on-the-street,  or  Air  Force  developer.  As  an 
example  of  how  complex  video  production  is,  consider  the  lighting  requirements  for  a  scene. 
Whether  to  use  a  single  light  source,  a  two-point  setup  with  key  light  and  back  light,  or  a  classic 
three-point  setup  using  fill  light  is  not  the  kind  of  decision  that  the  typical  Air  Force  developer 
knows  how  to  make.  Another  question  is  what  kind  of  lighting  to  use  to  create  greater  contrast 
and  dimension  and  when  to  apply  such  effects.  An  audio  producer  works  with  the  video 
producer  to  coordinate  talent  and  recording  equipment.  Knowledge  of,  or  experience  with 
training  programs  or  documentary  work  is  desirable.  The  sound  effects/music  coordinator  finds 
and  records  the  appropriate  music  and  sound  effects  for  the  application.  Specific  knowledge  of 
and  experience  working  with  copyrighted  material  is  highly  desirable.  All  production  specialists 
should  be  able  to  communicate  well  among  the  team  and  with  the  customer. 

Subject  Matter  Expert.  The  subject  matter  expert  (SME)  for  the  T-37  formation  flying 
lessons  was  from  Air  Training  Command's  338th  Operations  Training  Squadron  at  Randolph 
AFB.  He  provided  help  in  several  areas  in  addition  to  providing  flying  expertise.  The  SME  was 
also  experienced  in  instructional  design  and  had  a  doctorate  in  adult  and  vocation  education,  not 
typical  experience  or  education  for  a  SME.  Initially,  the  SME  helped  define  topics,  such  as 
learning  the  proper  system  of  visual  references  for  formation  flying  which  would  provide  an 
opportunity  to  exploit  the  visual  effects  of  multimedia  technology.  He  also  suggested  possible 
lessons  that  had  highly  visual  aspects  and  an  established  base  of  video  footage  which  could  be 
used  without  shooting  additional  sequences.  Close  interaction  between  the  multimedia  developer 
and  the  SME  elicited  a  wide  variety  of  useful  examples,  generalizations,  task  variations  and 
deviations.  In  addition  to  personal  training  and  flying  anecdotal  information  to  make  the 
courseware  more  concrete  and  realistic  to  users,  the  SME  also  provided  information  on  learner 
characteristics  including  a  description  of  their  skills  and  knowledges.  Some  discussion  centered 
around  what  the  learner  would  already  know  at  this  point  in  pilot  training.  These  decisions  were 
important  to  learner  control  decisions  in  instructional  design,  as  well  as  general  lesson  pacing. 
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Interactions  between  the  multimedia  developer  (a  composite  job  consisting  of  several 
skills  described  above)  and  SME  were  structured  differently  according  to  the  stage  of  lesson 
development.  Early  in  the  project  discussions  were  deliberately  unstructured  (e.g., 
brainstorming);  at  other  times,  the  SME  was  given  specific  written  assignments  which  produced 
highly  structured  feedback.  Several  knowledge  engineering  sessions  were  conducted  with  the 
SME  explaining  a  topic  to  the  team.  These  demonstrations  and  explanations  were  captured  on 
videotape  and  portions  were  later  creatively  incorporated  into  the  lesson.  At  other  times  the 
SME  brought  in  other  SMEs  who  were  expert  in  specific  areas  to  provide  their  input. 

Instructional  Design 

The  instructional  design  of  the  training  system  consists  of  two  aspects  —  content  and 
format.  The  content  of  the  T-37  Formation  Flying  training  system  consisted  of  five  modules  or 
books.  These  were:  ground  preparations,  taxi,  take-off,  formation  flying,  and  landing  procedures. 
The  format  of  the  lesson  modules  consisted  of  basic  instructional  pages,  supplemental  r  al  or 
tab  pages,  supplemental  information  accessed  via  hotwords,  and  tests.  When  combinec 
two  aspects  (content  and  format)  formed  the  top  level  design  of  the  courseware.  Instructional 
strategies  for  individual  lessons  were  selected  and  implemented  within  this  overall  training 
system  structure. 


Figure  1.  Courseware  Structure 


The  lesson  template  represents  the  training  system's  structure  used  to  organize  the  presentation 
elements  for  each  learning  objective.  A  hierarchy  was  established  for  each  learning  objective's 
procedural  and  conceptual  knowledge.  Ideas  and  concepts  often  overlapped  in  different  sections 
of  the  outline.  For  example,  the  underlying  information  is  the  same  in  Section  I  as  in  Section 
I.A.l,  however  the  emphasis  of  instruction  and  information  delivery  is  different  (see  Figures  1 
and  3).  Specifically,  required  information  in  the  application  was  presented  to  the  user,  and, 
although  the  user  was  not  required  to  view  the  supplemental  information  contained  on  a  tab  page, 
variations  of  required  data  were  available.  As  a  further  contrast,  this  information  was  presented 
by  a  female  voice  compared  to  the  male  voice  which  the  user  normally  heard  in  the  required 
information  pages. 


Figure  2.  Intuitive  User  Interface 


The  intuitive  user  interface  resembles  a  book  which  was  based  on  the  Pilot's  Abbreviated 
Flight  Crew  Checklist ,  T.0. 1T-37B-1CL-1,  (see  Figure  2).  Within  the  book,  information  was 
organized  into  sections  and  chapters.  Video  and  audio  clips  were  used  to  present  the  information 
on  each  page.  Text,  graphics,  and  animation  were  used  to  support  and  review  the  video.  Tabbed 
pages,  which  resembled  the  emergency  procedure  section  of  the  flight  crew  check  list,  provided 
supplemental  information.  The  presentation  of  lesson  information  was  intended  to  be  similar  to 
a  documentary  or  pre-flight  lecture.  Interactive  support  was  available  on  each  page  and  in  the 
assessment  or  testing  modules.  Testing  and  assessment  modules  were  designed  for  meaningful 
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interaction  and  specifically  supported  procedural  or  conceptual  learning  objectives.  For  example, 
various  animated  exercise  time  figures  are  used  to  motivate  students  to  test  their  knowledge  by 
showing  the  proper  sequence  for  taxiing,  etc.  In  this  type  of  assessment  students  highlight  and 
place  video  images  of  the  T-37  jet  in  the  correct  order,  receive  feedback,  and  proceed  on  to  the 
next  question.  In  addition,  extensive  audio  narration  was  incorporated  into  each  learning 
module.  In  fact,  the  audio  narration  leads  the  student  through  the  mainline  sequence  of  the 
lesson  and  also  in  the  supplemental  information.  The  only  places  narration  was  not  used 
extensively  was  for  help,  hotwords  and  in  assessment  or  testing. 

The  following  diagram  (Figure  3)  represents  the  hierarchy  of  information  delivery  for 
each  learning  objective.  Each  box  represents  a  simplified  view  of  the  training  system’s  screen 
and  available  branching  options.  Relevant  concepts  and  procedures  for  each  learning  objective 
are  made  available  and  presented  throughout  the  different  hierarchical  levels.  System  navigation 
is  available  through  a  simple  and  intuitive  user  interface.  This  interface,  allowing  access  to 
limited  choices  and  screen  buttons,  allows  users  to  become  comfortable  and  proficient  with 
system  navigation  quickly.  An  innovative  student-teaching  navigation  requires  the  student  to 
competently  explain  (or  leam)  concepts,  procedures,  or  instructional  goals  prior  to  moving  on 
through  the  system  to  the  next  group  of  objectives.  If  the  explanation  provided  in  the 
information  page  is  not  sufficient,  the  system  will  redirect  the  student  by  providing  access  to 
supplemental  information.  In  effect,  the  student  is  not  able  to  progress  through  the  system  unless 
there  is  articulated  an  acceptable  understanding  of  the  material. 

Figure  3.  Lesson  Hierarchy 
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Multimedia  Development 


Multimedia  combines  various  media  on  a  personal  computer  (PC)  in  a  single  digital 
format.  The  PC  allows  direct  access  to  the  digital  information  which  describes  each  multimedia 
element  in  the  application  such  as  full  motion  video,  high  quality  audio,  high  resolution  video 
images,  graphics,  animation,  and  text,  in  the  same  all-digital  format.  Multimedia  technology 
establishes  the  PC  as  a  powerful  training,  simulation,  and  cost  effective  delivery  platform.  DVI 
technology,  a  product  of  Intel  Corporation,  is  based  on  their  i750  PB/DB  chip  set.  These 
processors  are  dedicated  to  the  compression,  decompression,  display,  memory  management,  and 
integration  of  multimedia  elements.  They  are  contained  in  a  single  plug-in  board  supporting 
multimedia  applications  on  the  PC  (PS/2  and  Apple  Macintosh).  When  configured  with  a  snap- 
on  daughter  board,  DVI  can  digitize  full  motion  video,  still  images,  and  audio  directly  into  the 
PC.  Digitizing  full  motion  video  in  real  time  on  the  PC  is  a  technological  breakthrough. 
Although  the  resolution  is  approximately  VHS  quality,  this  is  a  cost-effective  method  to  obtain 
full  motion  video  for  most  computer-based  applications.  Future  generations  of  the  product  will 
certainly  introduce  algorithms  to  improve  real  time  video  capture  quality. 

Multimedia  Task  Hierarchy 

It  is  important  to  emphasize  that  all  activities  involved  with  the  T-37  formation  flying 
multimedia  interactive  courseware  development  were  performed  solely  for  research  purposes 
rather  than  as  part  of  a  full-scale  courseware  development  effort.  That  is,  repetitive  activities 
which  are  normally  performed  during  courseware  development,  were  sometimes  performed  only 
once  so  that  the  process  could  be  observed  and  documented.  Some  features  of  normal  lessons 
such  as  extensive  branching  or  remediation  loops  were  demonstrated  at  points  in  the  lessons  yet 
they  were  not  elaborated  throughout  all  the  demonstration  lessons.  Other  deviations  from  normal 
development  activities  were  explained  to  the  observer  as  they  occurred.  In  fact,  a  large  portion  of 
the  multimedia  development  team's  time  was  spent  verbalizing  and  explaining  the  underlying 
cognition,  decision-making  processes,  and  other  covert  behaviors  which  occurred  during  the 
normal  development  process. 

A  job  and  task  analysis  (Merrill  1987)  was  performed  to  provide  detailed  information 
about  the  organization,  structure,  critical  components,  and  sequential  relationships  between 
specific  jobs,  tasks,  and  duties  of  multimedia  development.  Multimedia  development  is  similar 
to  other  kinds  of  courseware  development.  However,  an  additional  process  not  likely  to  have 
been  accounted  for  prior  to  multimedia  applications  is  production.  Production  accounts  for  all 
issues  related  to  audio,  video,  still  image  and  their  capture,  e.g.,  analog  to  digital  conversion. 
While  production  is  a  major  part  of  multimedia  development,  the  process  of  producing  a 
multimedia  training  lesson  involves  several  other  components,  including:  1)  analysis,  including 
needs  assessment  and  evaluating  hardware  and  software  requirements;  2)  design,  including 
lesson  specification  and  designing  a  user  interface  and  system  navigation;  3)  implementation, 
including  system  prototype  development;  and;  4)  testing  and  debugging  the  system. 

Production  embodies  tasks  and  skills  that  have  been  applied  for  years  by  motion  picture, 
television,  and  radio  professionals.  Multimedia  development  integrates  several  of  these  skills 
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with  the  skills  of  traditional  instructional  development.  To  a  large  extent  the  success  of 
multimedia  development  depends  on  how  successfully  production  is  integrated  into  the 
traditional  development  process.  In  spite  of  its  many  new  aspects,  some  elements  of  production 
are  similar  to  traditional  courseware  development  processes.  For  example,  motion  video 
production  includes  similar  kinds  of  planning  activities  during  design.  Details  on  production  and 
other  multimedia  development  activities  can  be  found  in  the  task  analysis  and  model  of  the 
multimedia  development  process. 


in.  RESULTS 
Observational  Study 

A  review  of  the  observational  methods  used  in  this  study  shows  that  they  were  highly 
appropriate  and  effective  given  the  preliminary  nature  of  the  subject  matter  and  the  ultimate  goal 
of  the  task,  i.e.,  to  evaluate  methods  of  documenting  a  novel  training  process.  These  informal 
methods  produced  a  large  quantity  of  information  ranging  from  a  detailed  description  of  the 
multimedia  development  process,  the  technology  involved,  and  reactions  of  the  multimedia 
development  team  to  their  involvement  in  an  observational  study.  A  108  page  document  was 
produced  containing  the  raw  data  of  the  study  spanning  a  10-month  period.  These  data  are 
supplemented  by  a  series  of  audio  tapes  from  several  of  the  discussion  sessions,  and  two  verbal 
transcripts  from  structured  interviews. 

Temporal  and  sequential  relationships  between  CIs  are  potentially  extremely  useful  and 
are  shown  in  Appendix  A.  CIs  are  referred  to  and  discussed  individually  in  the  lessons  learned 
sections.  While  Flanagan's  (1954)  Cl  protocol  was  generally  followed,  the  DVI  usability  study 
acknowledged  many  CIs  which,  rather  than  being  discrete  and  obvious  critical  events,  appeared 
to  be  components  in  a  series  of  problems  requiring  solutions.  However,  each  statement  reflects 
an  important  event,  activity,  or  decision  point  which  ultimately  had  an  effect  on  the  multimedia 
development  process. 

Critical  incidents  relevant  to  multimedia  development  were  extracted  from  the 
observational  data  and  from  retrospective  structured  interviews  about  decision  points.  Eighty-one 
CIs  were  identified,  listed  in  temporal  order,  and  classified  according  to  types.  Events  were 
placed  in  one  of  two  broad  categories:  management  or  technical.  Management  CIs  reflected 
general  behavior,  problem  solving  and  creative  processes,  and  included  subcategories  of 
interpersonal  activities,  design,  scheduling  and  organization.  Technical  activities  focused  on  the 
development  team's  interaction  with  computer  software,  hardware  and  other  multimedia 
development  issues.  These  data  were  used  to  construct  Appendix  A  similar  to  the  Cl  chart  of 
Miles  &  Huberman  (1987),  listing  in  temporal  order  those  events  perceived  as  important, 
influential  or  decisive  in  developing  the  multimedia  lesson. 

Appendix  A  lists  CIs  according  to  their  reference  number,  date  of  occurrence,  brief 
description,  rationale,  and  effect.  For  example,  Cl  No.  53  is  described  "(Developer)  videotaped 
SMEs  demonstration  of  (formation  flying)  training  concepts."  The  rationale  was  "to  obtain 
original  and  realistic  instructional  footage."  The  effect  on  the  lesson  was  that  it  "provided 
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customized  training  system  details."  This  Cl  was  categorized  as  being  generally  relevant  to 
management  and  design  issues.  Appendix  A  also  lists  each  category  and  type  of  Cl.  For  all 
cases,  the  reference  date  is  either  the  actual  date  of  occurrence  or  it  refers  to  the  date  when  the 
observer  obtained  the  information  about  the  event  from  the  multimedia  development  team. 

The  effects  of  different  CIs  were  of  various  magnitudes.  For  example,  an  early  decision 
to  develop  DVI  in  the  relatively  new  DVIAVindows  environment  (Cl  No.  2)  was  critical  in 
planning  lesson  development  activities  as  well  as  shaping  production  capabilities. 
Standardization  of  transitions  between  video  clips  (Cl  No.  52),  by  comparison,  was  limited  to  a 
barely  noticeable  component  of  the  lesson.  The  arrangement  of  CIs  according  to  date  does  not 
reveal  any  outstanding  trends  or  associations.  It  appears,  however,  that  fewer  CIs  occurred  at  the 
beginning  of  the  research  compared  to  the  final  months  of  the  project,  and  occasionally,  several 
content-related  CIs  occurred  together.  For  example,  the  influence  of  a  Cl  such  as  (Cl  No.  47) 
"Little  or  no  documentation  for  DVI  software,"  affected  subsequent  progress  in  many  areas,  e.g.. 
Cl  No.  60,  "Problem  with  CGA  resolution  displaying  Targa  image,"  and  Cl  No.  71  "Bleeding 
around  DVI  window  border."  As  an  explanation  for  why  fewer  CIs  occurred  at  the  beginning  of 
the  project,  we  would  offer  the  fact  that  some  time  was  wasted  waiting  for  the  DVI  software  to 
be  readied  by  vendors. 

The  81  CIs  shown  in  Appendix  A  were  sorted  into  major  categories  and  sub-types.  There 
were  32  CIs  generally  classified  as  management,  divided  into  three  subcategories:  interpersonal 
(2  CIs),  design  (28  CIs),  and  scheduling/organization  (2  CIs).  The  49  technical  CIs  were  also 
divided  into  three  subcategories:  hardware  (7  CIs),  software  (17  CIs),  and  multimedia 
development  (25  CIs).  This  distribution  of  Cl  types  seems  to  match  the  perceived  pattern  of 
activities  related  to  management  of  design  and  technical  tasks.  A  relatively  large  number  of 
software  related  CIs,  such  as  Cl  No.  61,  "Problem  noticed  with  maintaining  detail  when 
shrinking  images,"  overlapped  with  multimedia  development,  even  though  they  were  separately 
classified.  Some,  such  as  Cl  No.  44,  "Development  team  was  unable  to  capture  still  image," 
contained  elements  of  both  design  and  technical  issues  depending  on  whether  it  was  categorized 
according  to  its  cause  or  effect.  The  most  common  source  of  overlap  was  in  the  technical 
category  between  software  and  multimedia  development,  which  was  the  area  the  multimedia 
development  team  appeared  to  pay  most  attention  to.  In  the  context  of  background  knowledge 
and  expertise  on  doing  multimedia  in  the  Windows  environment  and  availability  of  tools,  these 
CIs  appear  to  accurately  reflect  the  nature  of  the  most  important  issues  affecting  the  multimedia 
development  process. 

Multimedia  Job/Task  Analysis 

As  part  of  this  study  we  conducted  a  prototypical  task  analysis  of  the  multimedia 
development  process  including  multimedia  development  and  video  production.  Part  of  the  task 
analysis  was  a  result  of  the  observational  study.  After  the  multimedia  development  effort  was 
completed,  we  also  debriefed  the  multimedia  development  expert  to  elicit  from  him  those  tasks 
performed,  the  skills  and  knowledge  required  in  performance  of  multimedia  development,  and 
indications  of  what  elements  of  multimedia  development  are  most  affected  by  individual 
creativity.  As  a  result  of  debriefing  the  multimedia  expert,  we  asked  him  to  develop  a  task  listing 
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of  the  various  activities  involved  in  multimedia  development.  The  results  of  this  task  are  found 
in  Appendix  B:  Multimedia  Job  and  Task  Analysis. 


Appendix  B  contains  raw  data  which  has  only  been  edited  superficially  to  eliminate 
redundancies  whenever  possible  and  to  standardize  the  data  along  the  lines  of  traditional  Air 
Force  ISD.  The  numbering  system  used  reflects  that  provided  by  the  multimedia  expert,  and  in 
no  way  relates  to  the  numbering  system  used  in  the  Multimedia  Development  Model  found  in 
Appendix  C,  although  both  of  these  appendices  are  loosely  related.  The  task  data  serve  two 
purposes:  1)  it  provides  an  initial  look  at  multimedia  development  skills  as  they  relate  to 
interactive  courseware  development,  as  such  it  can  form  the  foundation  for  additional  research 
into  the  multimedia  development  process;  2)  it  formed  the  basis,  along  with  the  critical  incidents 
derived  from  the  observational  study,  for  the  development  of  a  model  of  the  multimedia 
development  process  as  it  fits  into  the  ISD  process  (see  Appendix  C). 

Ideally,  both  the  task  analysis  data  and  the  model  of  multimedia  development  should  have 
been  available  throughout  the  study  for  comparison  with  the  actual  data  and  observations,  but  as 
this  was  an  initial  study  of  multimedia  the  analysts  did  not  have  either  document  to  refer  to. 
These  appendices  now  provide  a  foundation  for  further  study  of  the  multimedia  development 
activities.  Ultimately,  additional  studies  would  result  in  additional  detail  for  both  tasks  and 
model  components,  thereby  providing  an  even  stronger  basis  for  extracting  multimedia  principles 
and  guidelines  for  novices. 


Lessons  Learned 

Development  of  the  prototype  T-37  training  application  using  DVI  technology  was  extremely 
challenging  from  both  a  hardware  and  software  perspective,  requiring  a  broad  range  of  technical 
skills,  knowledge,  and  problem  solving  ability.  As  this  newly  emerging  technology  becomes 
more  affordable,  rapid  changes  and  innovations  in  both  hardware  and  software  are  being 
introduced  almost  daily.  In  fact,  during  the  1 1 -month  period  of  this  project,  several  new 
advances  were  announced  and  placed  on  the  marked  that  are  expected  to  significantly  influence 
the  direction  of  multimedia  technology.  The  multimedia  development  team  encountered 
numerous  technical  challenges  while  working  on  the  lessons.  These  are  briefly  listed  in  the 
critical  incidents  (Cl)  chart  (Appendix  A)  and  more  thoroughly  discussed  in  this  section. 
Technical  challenges  are  described  in  terms  of  their  specific  problem(s),  solutions,  and  proposed 
alternative  approaches. 

Equipment  Selection 

A  critical  consideration  in  establishing  a  successful  multimedia  training  program  is  determining  a 
hardware  onfiguration  to  satisfy  requirements.  The  multimedia  development  team  researched 
two  options  for  obtaining  hardware:  1)  purchase  a  turn-key  system,  or;  2)  configure  and  build  a 
system.  The  team  decided  to  assemble  their  own  DVI  system  in  order  to  configure  a  faster 
system  with  greater  hardware  capacity  for  less  cost  (Cl  17).  Research  on  hardware  components 
to  determine  their  compatibility  with  the  Action  Media  II  boards  afforded  the  team  additional 
knowledge  and  experience.  They  also  found  that  this  new  technology  was  accompanied  by 


virtually  no  documentation  and  little  information  for  guidance  (Cl  47).  Hardware  purchases  were 
meticulously  investigated  and  compared  to  avoid  compatibility  problems.  An  alternative  strategy 
was  to  purchase  a  turn-key  system,  such  as  ones  offered  by  Avtex  or  Paragon.  Although  turn-key 
systems  have  an  initial  higher  cost,  all  hardware  components  undergo  compatibility  testing  and 
are  delivered  ready  to  use. 

An  incompatibility  problem  surfaced  between  the  team's  custom  configured  DVI  development 
PC  (Gateway  2000)  and  the  Air  Force's  turn-key  Dell  DVI  PC  (Cl  68).  For  example,  graphic 
images  that  displayed  the  correct  colors  on  the  team's  PC  displayed  differently  on  the  Air  Force 
equipment.  Missing  colors  on  the  Air  Force  PC  created  a  random  pixelating  effect  through  the 
image.  This  new  problem  was  identified  after  a  previous  incompatibility  with  the  Air  Force 
machine,  and  was  resolved  by  replacing  the  installed  16  color  VGA  video  driver  with  a  256  color 
VGA  video  driver  that  was  downloaded  from  the  Dell  Computer  bulletin  board.  Both  computers 
were  able  to  support  256  colors,  the  maximum  number  of  colors  able  to  be  displayed  through  the 
Action  Media  II  board  in  Windows. 

The  team  discovered  a  conflict  between  the  Power  On  Self  Test  (POST)  address  in  the  Action 
Media  II  boards  and  the  on-board  Basic  Input  Output  System  (BIOS)  address  of  the  Adaptec 
Small  Computer  System  Interface  (SCSI)  controller  board.  During  the  system  boot  process, 
initialization  code  from  both  boards  were  loaded  to  the  same  system  memory  location.  These 
addresses  were  pre-set  at  the  factory,  although  they  often  need  to  be  re-jumpered  to  avoid 
conflicts.  In  this  case,  the  address  jumpers  on  the  Adaptec  board  were  changed  to  load  to  another 
address.  The  Interrupt  ReQuest  line  (IRQ)  settings  that  peripheral  devices  used  to  inform  the 
computer  of  an  impending  action  that  needs  to  be  serviced  did  not  conflict  with  the  other  IRQs 
and  did  not  change. 

Display 

During  the  initial  set-up  of  the  hardware  and  software,  the  team  experienced  a  problem  with  the 
monitor  not  working  correctly  when  displaying  DVI  videos  on  the  screen  (Cl  76).  The  monitor 
would  turn  green  following  completion  of  a  motion, video  application  and  the  system  would 
sometimes  lock-up.  The  monitor  being  used  was  a  new  15-inch  multiscan,  non  interlaced  analog 
color  display  which  used  the  Flat  Square  screen  technology.  Special  properties  of  this  monitor 
were  described  to  the  system  via  a  monitor  command  included  in  the  AUTOEXEC.BAT  file.  To 
resolve  the  monitor  problem,  the  command  was  removed  from  the  file  and  the  VGA  card  was 
allowed  to  determine  the  monitor  default  settings. 

Asvmetrics  Capture  Software 

One  prominent  limitation  was  the  Asymetrics  capture  software;  this  software  proved  to  be 
inadequate  to  meet  the  requirements  of  DVI  training  system  development  (Cl  34).  Although 
three  levels  of  resolution  quality  could  be  optimally  selected,  the  best  level  (e.g.,  highest  bit 
capture  rate  per  second)  did  not  produce  adequate  video  quality  from  the  poor  quality  tapes 
provided  for  the  application.  The  digitizing  process  did  not  recognize  changing  sync  levels,  and 
as  a  result  produced  flickering,  bleeding,  and  picture  distortions  as  the  team  integrated  captured 
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video  from  various  tape  sources.  Additionally,  there  was  no  pause  capability  to  selectively  play 
and  capture  segments.  This  required  the  team  to  edit  video  prior  to  the  capture  process. 

Currently,  video  can  only  be  captured  in  a  160  pixel  by  120  pixel  resolution  window;  images 
cannot  be  scaled  up  or  down  and  parts  cannot  be  added  or  taken  away.  This  is  a  limitation  of  the 
Intel  video  drive,  although  there  are  plans  to  upgrade  the  drive  so  that  it  can  eventually  support 
320  x  200  resolution.  Higher  resolution  allows  for  larger,  crisper  images. 

The  team  encountered  another  problem  during  the  video  capture  and  digitizing  process.  The 
capture  program  would  abort  if  the  AVS  (capture)  file  exceeded  the  113-116  MB  range. 
Asymetrics  reported  that  this  problem  had  not  been  encountered  during  the  Beta  testing  of  this 
software.  However,  after  the  team  provided  a  detailed  description  of  their  capture  process, 
Asymetrics  was  able  to  recreate  the  problem.  Asymetric's  final  Beta  release  of  the  capture 
software  corrected  the  aborting  problem. 

Color  Palette 

A  table  of  color  information,  called  a  color  palette,  exists  in  Windows  that  defines  and  sets  the 
system  colors.  If  these  colors  are  not  identical  between  microcomputers  or  applications,  the 
result  may  be  a  loss  of  colors  in  bitmap  images  when  displayed  on  other  computers.  This  appears 
to  be  the  problem  the  team  experienced  with  the  Toolbook  bitmap  images.  Images  captured 
(digitized)  on  one  computer  displayed  the  graphic  correctly  on  that  computer,  but  when  uploaded 
to  the  other  DVI  development  computer,  the  graphic  frequently  displayed  a  pixelating  effect. 

The  team  also  noticed  that  colors  for  bitmap  images  were  frequently  changed  to  a  different  tint 
on  screens  which  contained  multiple  images.  Additionally,  the  bitmap  image  became  distorted 
each  time  the  user  moved  to  a  new  page  of  the  Book.  In  256  color  bitmap  files  (for  graphic 
images)  the  color  palette  is  stored  with  the  graphic.  The  color  palette  is  used  to  determine  and  set 
the  colors  of  the  displayed  image.  If  multiple  images  are  placed  on  a  single  page  in  Toolbook, 
the  color  palettes  must  be  identical,  or  no  more  than  256  colors,  otherwise  images  may  contain 
undesirable  colors.  Similar  undesirable  effects  occur  if  color  palettes  conflict  with  the  graphics 
on  other  pages  of  the  Book.  As  the  pages  are  displayed,  conflicting  palettes  force  a  re-paint  of 
the  screen  each  time  a  different  palette  is  used.  The  result  is  a  distortion  of  the  image  and  a  brief 
delay  each  time  a  new  page  is  displayed.  The  solution  is  to  ensure  the  same  color  palette  is 
applied  to  all  graphic  images  during  the  creation  process. 

Bitmap  Images 

There  was  a  common  problem  moving  (e.g.,  dragging  and  dropping)  bitmap  images  on  the 
display  screen  with  a  mouse  device.  Frequently,  residue  in  the  form  of  lines  or  dots  appeared  on 
the  screen  as  the  image  moved.  The  team  investigated  this  problem  and  proposed  some  possible 
solutions.  First,  all  drag  and  drop  images  were  set  to  drawdirect.  This  eliminated  the  residue 
problem,  but  caused  the  image  to  noticeably  flicker  when  it  was  dragged.  This  solution  was  not 
implemented  since  the  flickering  was  more  irritating  than  the  residue.  Drawdirect  images 
flickered  when  they  were  moved  because  they  were  composed  directly  on  the  screen  rather  than 
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off-screen  in  memory.  Second,  the  team  refreshed  the  background  each  time  the  image  was 
moved.  This  approach  removed  the  residue  once  the  movement  of  the  image  was  completed. 
However,  the  residue  still  appeared  while  the  image  was  in  motion.  This  solution  of  refreshing 
the  background  also  introduced  unnecessary  processing  overhead.  The  team  noticed  that  the 
residue  problem  did  not  exist  on  the  Air  Force  DVI  development  computer.  It  appeared  that  the 
VGA  driver  for  the  team's  computer  caused  the  residue  to  remain  on  the  screen.  To  fix  this 
problem,  the  team  downloaded  a  new  VGA  driver  from  the  Diamond  SpeedStar  bulletin  board. 

Audio  Level 

Two  weeks  after  buildii.^  he  production  digital  video  files,  the  team  noticed  a  significant  drop  in 
the  level  of  audio  volume  on  certain  frame  sequences  in  one  of  the  files.  This  file  was  digitized 
and  loaded  to  hard  disk  storage  at  the  production  studio  by  playing  the  master  analog  tape  on  the 
professional  analog  equipment  and  capturing  (digitizing)  it  directly  to  the  DVI  development 
computer's  hard  disk.  At  first  the  team  thought  that  the  decline  in  audio  may  have  resulted  from 
a  damaged  audio  track  in  the  file.  Their  initial  investigation  showed  the  file  to  be  intact  with  no 
degradation  to  either  the  audio  or  video.  Using  special  DOS-based  utilities  for  verifying  the 
format  of  AVS  files,  the  team  was  able  to  demonstrate  that  the  file  contained  no  abnormalities 
Further  investigation  using  an  analog  meter  to  measure  the  audio  decibels  (DB)  revealed  that  the 
master  tape  had  significant  intermittent  drops  in  audio  volume  due  to  inconsistencies  in  the 
production  process  (Cl  67).  The  remedy  was  to  re-digitize  the  problem  file  and  boost  the  audio 
through  an  audio  mixer/amplifier.  The  same  results  could  also  have  been  obtained  through  a 
manual  volume  adjustment  process  using  an  audio  compressor  limiter  device. 

Scripting 

Toolbook  was  the  authoring  environment  used  to  develop  the  lessons.  OpenScript  was  the 
object-oriented  language  used  to  produce  the  script  routines  to  control  the  behavior  and 
properties  of  objects  within  the  application  All  applications  created  within  Toolbook  were 
event-driven,  such  as  pressing  the  replay  button  or  requesting  the  index,  with  events  in  tum 
triggering  a  message.  Each  message  in  tum  is  sent  to  an  object  which  may  or  may  not  have  an 
associated  script  programmed  to  respond  to  it. 

Management  k$ues 

The  development  team  which  worked  on  this  project  was  very  small.  In  fact,  for  the  first  several 
months  while  waiting  for  software  issues  to  be  resolved  the  multimedia  developer  alone  worked 
with  the  SME.  The  multimedia  development  process  would  have  been  different  if  there  had  been 
an  interactive  courseware  production  schedule  to  meet  with  a  larger  development  team  involved 
from  the  beginning  of  the  project.  For  example,  the  team  would  have  had  many  overlapping 
tasks  and  different  individuals  would  be  working  in  parallel  on  the  same  task.  In  the  case  of  this 
research,  the  number  of  personnel  participating  was  purposely  limited.  The  focus  of  the  study 
was  on  multimedia  development  and  production,  and  consequently  the  team  was  limited  to  only 
those  necessary  to  contribute  to  the  research  goal. 
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flbssnational  Bssfiarsh  Issues 


This  preliminary  study  focused  on  observing  the  behavior  of  a  multimedia  expert  and 
development  team  working  on  a  small  research  project  that  may  or  may  not  be  representative  of 
all  multimedia  development  efforts  (Perez  and  Neiderman  1992).  Detailed  analyses  of 
observations  of  the  multimedia  development  processes  produced  data  on  temporal  and  structural 
relationships  between  tasks  and  personnel.  Although  there  is  a  lot  of  variability  in  terms  of 
personal  experience  and  problem  solving  strategies,  some  proposed  models  of  structure  and 
process  may  be  representative  among  individuals,  situations,  and  diverse  behavioral  phenomena 
(Greene  1988).  Further  observational  research  in  this  area  will  help  clarify  relevant  multimedia 
development  decision  making  behaviors  for  integration  into  computer-based  courseware 
authoring  aids. 

Two  other  aspects  of  conducting  observational  research  should  be  considered.  First,  the 
presence  of  an  observer  can  affect  the  subject's  behavior  as  well  as  the  interaction  of  team 
members.  A  short  period  when  the  observer  does  not  collect  data  but  allows  the  subject(s)  to 
accommodate  to  the  observer's  presence  is  useful  at  the  beginning  of  the  project.  It  was  also 
appropriate  at  this  time  for  the  observer  to  explain  her  role  in  the  research  project  and  to  describe 
what  was  expected  from  participants.  The  observer  stressed  that  factial  data  would  be  collected; 
no  judgments  would  be  made  about  the  value  or  utility  of  behavior.  There  was  no  right  or  wrong 
multimedia  development  process.  It  was  emphasized  that  team  members  were  not  expected  to 
perform  for  the  observer.  The  team  was  expected  to  make  mistakes  and  the  observer  was 
expected  to  document  exactly  what  the  team  did. 

Second,  it  sometimes  appeared  to  the  multimedia  development  team  that  the  observer  asked 
numerous  basic,  and  sometimes  redundant  questions.  Without  the  observer,  team  members 
would  not  have  had  to  stop  what  they  were  doing  to  answer  questions.  Sometimes  the  questions 
required  answering  with  a  lot  of  details  or  diagrams.  This  helped  document  the  multimedia 
development  process.  The  observer  also  frequently  asked  the  multimedia  development  team  to 
verify  the  observations.  Whenever  there  was  a  lot  of  activity,  different  activities  occurring 
simultaneously,  or  activities  occurring  when  the  observer  was  not  present,  the  observer 
questioned  the  participants  about  what  they  did.  This  was  occasionally  viewed  as  an  interruption, 
and  extraneous  to  the  project  as  the  team  was  focused  on  multimedia  development  not  research. 

It  was  helpful  for  the  observer  to  learn  as  much  as  possible  about  multimedia  development  and 
DVI  technology  before  the  project  started  in  order  to  reduce  the  number  of  background  questions 
and  focus  instead  on  the  processes  unique  to  the  T-37  formation  flying  interactive  courseware 
development. 

This  project  generated  numerous  data  for  one  observer  to  record  and  organize.  This  endeavor 
would  be  far  more  complicated  if  the  team  had  several  more  members  or  if  the  project  were 
larger  in  scope.  For  non-pilot  data,  participation  by  two  or  more  observers  would  be  useful  for 
increased  validity,  reliability,  and  effectiveness.  Additionally,  a  full  scale  production  project 
could  incorporate  more  labor-intensive  methods  such  as  protocol  analysis.  If  protocol  analysis 
were  used,  computer  programs  which  analyze  qualitative  data,  including  content  analysis  and 
coding,  would  help  streamline  and  standardize  these  tasks  (Richards  &  Richards  1991). 
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DYI  Technology 

As  an  enhancement  to  computer-based  training  technology,  multimedia,  and  specifically  DVI 
technology  as  a  delivery  platform  for  highly  visual  skills  and  tasks,  provides  many  new 
opportunities  for  conveying  information,  including  developing  and  delivering  training  (Luther 
1991).  These  are  characterized  by:  1)  increased  storage  capability  for  digital  data;  2)  ability  to 
randomly  access  these  data,  and;  3)  ability  to  electronically  produce  and  edit  audio  and  visual 
materials.  These  capabilities  have  important  implications  for  facilitating  teaching  and  learning. 
The  incorporation  of  video  into  interactive  training  systems  can  present  dynamic  information  to 
students  while  increasing  their  attention  and  potential  involvement  in  the  lesson.  Flexible  or 
random  access  to  information  offers  increased  opportunities  for  student  interaction  including 
branching,  assessment,  and  feedback. 

Hardware/Software  Configuration 

This  project  used  a  Windows-based  DVI  development  environment.  Hardware  and  software 
decisions  reflected  the  Laboratory's  goal  to  develop  on  a  system  compatible  with  potential  future 
Air  Force  computer-based  training  platforms,  including  Windows-based  AIDA. 


Table  1.  MPC  Standards 


CPU 

386SX  or  compatible  microprocessor  1 

RAM 

2MB 

Magnetic  Storage 

Floppy  drive,  30  MB  hard  drive  | 

Optical  Storage 

CD-ROM  with  CD-DA  outputs  | 

Audio 

DAC,  ADC,  music  synthesizer,  on-board  analog  mixing 

Video 

VGA  graphics  adapter,  VGA  color  monitor 

Input 

101  key  keyboard  (or  equivalent),  2-button  mouse 

I/O 

Serial  port,  parallel  port,  MIDI  I/O  port,  joystick  port 

Multimedia  platforms  can  be  divided  into  three  performance  levels:  one  level  is  the 
minimum  required  for  delivery;  the  next  level  is  a  high-powered  delivery  system;  the  third  level 
is  for  multimedia  development.  A  minimum  level  multimedia  PC  is  capable  of  sound  and 
graphics  and  is  usually  configured  with  a  CD  and  a  large  hard  drive.  However,  the  performance 
requirements  of  digital  video,  higher  fidelity  images,  sound,  etc.,  have  demonstrated  that  the 
Multimedia  PC  Council's  (MPC)  minimum  standards  for  multimedia  are  already  unrealistically 
limited.  The  MPC  approved  minimum  standards  for  a  DOS-compatible  machine  is  shown  in 
Table  1. 


The  hardware  suite  used  on  this  project  consisted  of  a  Gateway  2000  486DX  50  Mhz  with 
a  15-inch  SVGA  (NI)  monitor,  1.2.  GB  Seagate  SCSI  drive,  1.2  MB  5.25-inch  and  1.44  MB  3.5- 
inch  Epson  floppy  drives,  64k  SRAM  cache,  16  MB  DRAM,  16-bit  Adaptec  1442  ISA  disk 
controller,  120  MB  Archive  tape  back  up  drive,  Diamond  Speed  Star  video  board,  Sony 
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amplified  speakers,  CD-ROM  CDU-541,  Action  Media  II  delivery  board  (ISA)  and  Action 
Media  II  capture  module. 

Action  Media  II  Boards 

The  Intel  Action  Media  II  boards  are  designed  to  provide  a  broad  range  of  capabilities. 
The  Action  Media  II  boards  integrate  full  motion  video  (full  color,  full  screen  at  30  frames  per 
second  (fps)),  with  high  quality  still  images,  CD  quality  audio,  text,  animation,  and  graphics. 
However,  the  Intel  developed  driver  for  Windows  has  significant  limitations  that  severely  restrict 
DVI  development  capabilities.  For  example,  the  boards  are  capable  of  producing  stereo  audio, 
but  the  driver  restricts  audio  to  mono  only.  The  boards  are  capable  of  displaying  or  playing 
multiple  motion  video  files  subject  to  onboard  VRAM  constraints.  Windows  restricts  motion 
video  to  only  one  active  file  at  a  time,  although  this  limitation  does  not  apply  to  still  image  files. 
Multiple  still  image  files  may  be  open  and  actively  displayed  as  long  as  the  onboard  VRAM  is 
not  exceeded.  The  boards  are  capable  of  supporting  1024  by  768  by  16.8  million  colors,  but  the 
driver  limits  resolution  to  640  by  480  pixels  and  color  display  to  256. 

Action  Media  II  Hardware  Configuration 

The  Action  Media  II  board  can  be  configured  three  different  ways  as  shown  in  Figure  4. 
First,  an  external  RGB  overlay  cable  can  be  used  which  permits  operation  of  the  Action  Media  II 
board  with  a  single  video  monitor.  This  is  accomplished  by  connecting  outputs  of  the  Action 
Media  II  board's  VGA  connector  and  the  computer's  VGA  connector  to  a  single  monitor. 

Second,  connection  can  be  made  by  an  internal  VESA  cable  that  is  included  with  the  board.  This 
allows  for  a  single  monitor  setup  by  internally  connecting  the  VESA  cable  between  the  VESA 
connector  on  the  Action  Media  board  and  the  VESA  connection  on  a  VGA  graphics  board  or  the 
VESA  connection  of  the  built  in  VGA  of  the  computer's  mother  board.  This  provides  the  ability 
to  mix  VGA  graphics  with  DVI  video  on  the  same  monitor.  The  third  configuration  involves 
two  VGA  monitors.  One  monitor  is  used  for  the  VGA  display,  including  text,  graphics,  and 
operation  Of  the  authoring  language;  the  other  monitor  is  used  only  for  the  display  of  DVI  video. 

The  configuration  used  influences  the  development  of  an  application.  For  example,  if  an 
application  is  developed  for  a  single  monitor  system,  and  is  later  played  back  on  a  two  monitor 
system,  the  DVI  video  will  be  displayed  in  a  window  on  the  second  monitor  and  the  application 
will  have  a  blank  window  where  die  video  would  normally  be  displayed.  Some  hardware 
vendors  informed  the  multimedia  development  team  that  they  could  not  get  the  single  monitor 
external  cable  configuration  to  operate  and  preferred  the  relatively  easy  set  up  of  the  two  monitor 
system.  This  information  did  not  match  the  findings  of  the  multimedia  development  team  since 
they  found  no  problems  with  their  single  monitor  external  cable  system.  One  possible  reason 
could  be  that  in  order  for  video  keying  to  operate  correctly,  the  authoring  software  must  support 
video  keying  and  settings  within  the  AVK.INI  files  and  must  reflect  this  set  up.  This  is  usually 
taken  care  of  during  the  installation  of  software  and  set  up  of  the  Action  Media  n  board. 
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A  limited  range  of  software  is  available  for  Action  Media  n  DVI  boards.  One  such 
package  is  Video  for  Windows.  Using  the  Action  Media  II  boards,  this  product  allows  the 
capture  of  audio  and  video  at  15  fps  in  a  120  x  160  pixel  window  to  a  file  which  is  called  audio 
video  interleave  (.AVI).  This  format  shows  indications  of  becoming  the  new  standard  based  on 
the  availability  of  new  software  products  and  it  is  support  by  Microsoft.  The  software  allows 
cutting  and  pasting  of  audio  and  video  files  using  the  standard  Windows  clipboard  function  that 
Windows  uses  for  other  applications  such  as  word  processing  and  paint  programs.  This  is  called 
dynamic  data  exchange  (DDE).  Currently,  there  are  some  disadvantages  of  Video  for  Windows 
since  audio  can  be  captured  from  the  Action  Media  board  but  must  be  played  back  on  a  standard 
multimedia  PC  audio  board  (e.g.,  Sound  Blaster).  When  an  audio  board  is  used  with  the  Action 
Media  board,  one  of  two  conditions  must  be  met:  1)  a  second  pair  of  self-powered  speakers  is 
used—  one  pair  for  the  Action  Media  board  and  one  pair  for  the  sound  board;  2)  a  sound  board 
which  accommodates  an  auxiliary  line  level  input  on  the  board  is  used  so  the  Action  Media's 
audio-out  can  be  looped  through  the  sound  board  and  played  out  the  sound  board's  speakers. 
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Figure  5.  Sound  Board  Connections 


There  are  several  ways  to  edit  audio  and  video  for  DVI.  Off-line  editing  using  production 
editing  equipment  creates  a  completed  sequence  for  later  digitization,  which  includes  all  the 
prescribed  cuts,  fades,  transition,  highlights,  graphics,  and  text.  The  digitized  file  can  then  be 
played  back  in  the  multimedia  application  as  a  whole  or  segments  of  the  file  can  be  played  (e.g., 
frame  to  fiame).  This  process  currently  produces  the  best  results  since  Windows-based  DVI 
editing  software  is  not  yet  widely  available.  However,  this  type  of  editing  can  be  very  costly  if 
equipment  must  be  purchased  or  a  studio  rented.  Editing  suite  rental  can  range  from  $25.00/bour 
to  $400.00/hour  or  more  depending  on  the  specific  equipment  and  types  of  personnel  needed. 

The  Air  Force  has  several  editing  production  facilities,  such  as  Combat  Camera  at  Lackland 
AFB.  Once  again,  a  reminder  that  the  possession  of  a  technology  does  not  provide  the  capability 
to  use  it  effectively.  When  utilizing  such  facilities  to  develop  a  training  program  the  Air  Force 
would  be  wise  to  ensure  that  a  sound  instructional  design  has  been  accomplished  first  which 
specifies  exactly  what  is  needed.  These  facilities  may  also  have  access  to  stock  video  footage 
that  may  be  useful  in  creating  many  multimedia  applications.  A  word  of  caution  about  using 
such  stock  video  footage  in  an  application;  if  the  application  is  produced  with  less  than  master 
quality  or  at  least  first  generation  footage,  the  quality  of  the  video  is  reduced  dramatically. 

Editing  can  also  be  accomplished  by  digitizing  clips  and  pieces  of  audio  and  video  from 
video  cassette  recorders  (VCRs),  audio  tape  recorders  ( ACRs),  and  cameras.  These  segments  can 
be  cut  and  pasted  together  to  create  audio  visual  (AV)  files  without  special  effects.  Such  images 
would  simply  cut  from  screen  to  screen  unless  the  captured  material  had  existing  special  effects. 
During  this  multimedia  development  process,  the  only  editing  software  available  was  a  DOS- 
based  editor  that  required  a  repetitive  process  of  calculating  the  start  and  stop  frames  of  a 
sequence,  copying  them  into  new  files,  exiting  the  editor,  entering  Windows,  and  playing  the  file 
to  see  if  it  appeared  correctly.  However,  the  error  checking  capabilities  of  this  method  were  poor 
and  could  cause  a  system  failure  if  frames  were  not  correctly  copied.  After  completing  the 
demonstration  lessons,  Asymetrics  released  an  editor  utility  that  allows  a  programmer  to 
concatenate  audio  and  video  tracks  into  one  file,  however,  this  utility  does  not  permit  cutting  and 
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pasting  separate  audio  and  video  tracks,  other  software  packages  must  be  utilized  to  perform 
these  activities. 


Figure  6.  Audio/Video  Log 


Sample  Screen:  Video  Log 
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Audio  Video  Capture  Processes 

Capture  is  the  process  of  digitizing  audio  and  video  sequences  from  a  source  such  as  a 
VCR,  ACR,  camcorder,  laser  disk  player,  live  camera,  or  microphone.  Video  is  played  from 
these  devices  at  30  fjps,  standard  for  full  motion  video.  A  frame  of  video  contains  a  sampling  of 
an  instant  of  live  video.  Samples  are  stored  in  an  analog  format  (VCRs  and  ACRs)  or  in  a  digital 
format  (PC  hard  drives,  CD-ROMS,  and  some  high  end  D2  format  VCRs).  Several  capture 
methods  were  used  during  this  study.  Asymetric's  CAPTURE  and  PLAYER  routines  were 
primarily  used  for  the  production  of  the  demonstration  lessons.  In  conjunction  with  the  capture 
process,  the  multimedia  development  team  created  a  simple  Toolbook  application  which  the 
programmer  used  to  log  all  of  the  stock  audio  and  video  footage  (see  Figure  6).  This  information 
provided  the  capability  to  search  for  a  particular  scene  or  event,  retrieve  the  tape  counter  index 
number  of  the  tape,  and  queue  it  for  digital  capture.  Once  the  material  was  digitized,  the  digital 
reference  information  was  entered  in  the  Toolbook  logging  application  for  future  reference. 
Since  there  was  no  Pause  command  on  the  CAPTURE  routine,  captured  scenes  may  need  to  be 
concatenated  together.  This  was  accomplished  through  a  DOS  utility  that  was  downloaded  from 
the  INTEL  bulletin  board  service. 
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PYI  Full  Motion  V fflra  Comp 


lull'll 


Using  DVI  technology  to  edit  full  motion  video  involves  making  a  decision  about  what 
kind  of  full  motion  video  compression  to  use-  Real  time  video  (RTV)  or  professional  level  video 
(PLV).  The  capture  routine  software  digitized  each  file  in  real  time  on  the  development  system, 
however  RTV  is  lower  in  resolution  and  lower  in  frame  rate  than  PLV.  Alternatively,  in  order  to 
produce  PLV,  the  team  would  have  had  to  produce  a  video  master  and  send  it  to  a  professional 
compression  facility  which  uses  a  specialized  procedure  to  covert  to  PLV.  This  is  a  relatively 
quick  process  but  charges  are  about  $250  per  running  minute.  The  multimedia  development 
team  felt  that  RTV  produced  acceptable  results  for  their  prototype  lesson  development  (Cl  57). 

Instructional  Design  Issues 


A  deliberate  effort  was  made  to  limit  the  number  of  prototype  lesson  modules  (Cl  14). 
Three  learning  modules  were  originally  planned  for  the  demonstration.  One  of  the  proposed 
learning  modules  was  rejoins,  a  highly  complex  skill  to  portray.  After  viewing  the  existing  Air 
Force  videos  on  rejoins,  the  multimedia  development  team  felt  that  the  distant  views  of  flying 
aircraft  would  not  digitize  well.  Therefore,  the  team  decided  to  concentrate  on  the  modules  for 
fingertip  formation  and  airwork  (Cl  16),  for  a  more  intensive  effort  on  fewer,  but  more  robust 
modules  which  took  better  advantage  of  DVI's  capabilities. 


In  a  unique  approach  lessons  were  prototyped  directly  on  the  computer  screen  instead  of 
going  through  several  iterations  using  storyboards.  The  main  reason  for  this  was  the  fact  that  a 
single  multimedia  developer  was  working  on  the  project,  and  it  was  more  efficient  to  go  from  the 
ideas  in  his  head  directly  to  the  system  prototypes  (Cl  36).  This  method,  however,  was  not 
particularly  useful  for  discussions  with  the  SME  who  was  oriented  towards  the  use  of  traditional 
ISD  and  preferred  to  discuss  previous  and  current  versions  of  the  lesson  by  comparing 
storyboards. 

During  the  entire  multimedia  development  process  the  team  created  and  maintained  a 
comprehensive  table  showing  the  origin,  destination,  and  status  of  each  multimedia  element,  e.g., 
graphic,  full  motion  video,  etc.  (Cl  63).  This  facilitated  programming  and  playback  of  the  lesson 
frames  and  working  on  the  lesson  navigation  system.  Without  such  a  database  of  multimedia 
elements  there  would  have  been  confusion  regarding  audio,  video,  graphics  and  animation  files 
used  in  the  application. 

The  initial  plan  for  displaying  full  motion  video  was  to  display  graphics  on  the  screen  at 
certain  points  in  the  audio  track,  for  example,  an  arrow  comes  up  with  a  key  word  to  show  the 
location  of  visual  references.  However,  the  team  discovered  that  this  could  not  be  accomplished 
during  audio  capture  (Cl  59).  Instead,  they  overcame  the  problem  by  using  the  technique  of 
specifying  a  frame  number  at  which  to  insert  a  graphic  overlay. 
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Model  of  Multimedia  Development 


One  of  the  goals  of  this  project  was  to  determine  what  a  multimedia  expert  does  in  the 
course  of  developing  a  multimedia  application,  and  to  extract  from  these  activities  principles  or 
guidelines  which  could  be  used  by  Air  Force  instructional  developers  in  working  with  the 
medium.  Several  steps  performed  during  the  project  have  provided  a  large  amount  of  raw  data 
about  multimedia  and  the  multimedia  development  process.  In  developing  a  model  of 
multimedia  development,  we  have  attempted  to  provide  some  organization  and  logic  to  this  data 
so  that  it  can  be  applied  to  the  development  of  principles  and  guidelines  as  mentioned  earlier. 

The  model  stands  alone  as  a  representation  of  the  kernel  activities  of  multimedia  development. 
However,  it  should  be  viewed  as  a  first  attempt  at  describing  what  multimedia  development  is  all 
about.  Further  research  is  necessary  to  test  whether  this  model  1)  represents  true  generalizations 
about  the  activities  of  all  or  most  multimedia  experts,  2)  whether  it  is  functional  in  the  Air  Force 
instructional  development  environment,  and;  3)  whether  all  or  portions  of  the  model  are  capable 
of  automation  in  such  applications  as  AIDA. 

The  multimedia  development  model  provided  in  Appendix  C  is  a  highly  dynamic  one  in 
that  it  describes  each  major  function  or  activity  performed  in  multimedia  development,  and  given 
the  right  input  to  a  functional  component  or  process  it  can  generate  or  predict  the  output(s)  of  the 
component.  The  model  differs  from  traditional  surface  models  which  only  superficially  represent 
activities,  relationships  or  elements  of  a  system.  Surface  models  normally  provide  some 
graphical  representation  of  a  system  and  discuss  in  general  terms  the  functionality  of  the  system. 
This  model  represents  a  deep  structure  of  the  multimedia  development  process.  Activities, 
inputs,  outputs,  relationships,  and  rules  which  guide  the  operation  of  the  system  are  described  to 
a  level  which  provides  adequate  depiction  of  the  system  to  allow  for  specific  inferences  to  be 
made  about  how  multimedia  development  takes  place. 

In  modeling  multimedia  development  activities  we  have  couched  them  in  terms  of  the 
greater  ISD  process  within  which  they  would  normally  fall  in  the  course  of  Air  Force 
instructional  development.  The  top  level  of  the  model  depicts  the  traditional  ISD  model.  Only 
when  exploding  the  ISD  process  into  its  individual  phases  do  we  begin  to  describe  the  impact  of 
multimedia.  Certainly,  multimedia  can  have  an  impact  on  components  of  ISD  other  than 
development,  c.g.,  it  should  be  taken  into  account  when  selecting  media,  designing  the  training 
system,  etc.,  but  for  this  study,  we  focused  on  providing  an  in  depth  description  of  the 
multimedia  development  process  (including  production).  Therefore,  the  model  hierarchy  chart 
indicates  that  data  flow  diagrams  have  only  been  elaborated  for  the  major  function  Develop 
Training  and  below.  According  to  this  hierarchy  chart  the  reader  can  see  that  the  other  major 
components  of  ISD  to  be  described  elsewhere. 

Some  explanation  should  be  provided  as  to  why  we  chose  top-down  structured  analysis 
and  system  design  techniques  to  model  multimedia  development.  Several  reasons  can  be 
offered:  1)  one  important  reason  is  that  it  provides  a  ready  set  of  tools  for  modeling  systems,  i.e., 
data  flow  diagrams  with  built-in  diagramming  conventions,  process  descriptions,  and  a  data 
dictionary,  2)  for  modeling  purposes  ISD  or  multimedia  development  can  be  considered  a  system 
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with  all  the  features  of  one,  and;  3)  any  attempt  to  automate  a  system  or  process  requires  detail 
which  most  other  modeling  techniques  do  not  provide. 

As  mentioned  earlier,  the  derivation  of  principles  or  guidelines  from  this  study  was  not 
the  only  goal.  If  automation  is  to  be  considered,  it  is  first  necessary  to  describe  the  system  in 
such  detail  that  a  determination  can  be  made  as  to  what  portions  can  be  automated  and  how  the 
automation  is  to  be  designed.  This  model  provides  readers  with  the  ability  to  make  some  of 
those  determinations.  When  viewing  the  model  for  automation,  particularly  in  terms  of  being 
incorporated  into  AIDA,  a  system  designer  must  bear  in  mind  that  AIDA  departs  from  the 
traditional  ISD  approach  to  designing  and  developing  courseware.  Such  artifacts  as  storyboards 
are  bypassed  by  AIDA  to  create  courseware  directly.  We  should  note  that  our  multimedia 
developer  also  bypassed  storyboarding  causing  some  concern  on  the  part  of  the  SME  until  he 
became  accustomed  to  seeing  concepts  translated  directly  into  visual  representations  without 
going  through  the  intervening  paper-based  steps.  These  factors  have  implications  for  how  his 
model  should  be  viewed.  In  other  words,  whenever  the  model  indicates  data  are  derived  from  a 
storyboard  to  perform  some  multimedia  process,  when  considering  incorporation  into  AIDA 
system  designers  will  have  to  remember  that  those  data  will  have  to  be  provided  by  some  other 
source. 


In  the  strictest  sense,  any  top-down  structured  analysis  would  describe  the  processes  of  a 
system  with  mini-specifications  written  in  structured  language  ultimately  aimed  at  automation  of 
the  system  or  some  of  its  components.  We  have  departed  from  that  convention  in  this  model  to 
provide  for  ease  of  reading  on  the  part  of  those  oriented  towards  ISD  rather  than  systems  design 
and  development.  We  have  provided  a  data  dictionary  which  lists  all  of  the  terms  used  in  the 
multimedia  development  model. 


IV.  CONCLUSIONS 
Behavior-Technology  Interactions 

Promising  technologies  for  training  are  emerging  in  highly  usable  forms.  Multimedia, 
specifically  DVI,  represents  important  new  technologies  because  it  provides  an  instructional 
designer  the  ability  to  develop  and  professionally  edit  compelling  and  informative  interactive, 
multimedia  lessons.  Unfortunately,  these  are  not  simple  tasks.  Considering  the  highly  complex 
processes  described  in  the  lessons  learned,  it  is  unlikely  that  Air  Force  instructional  designers 
have  the  technological  expertise  to  incorporate  multimedia,  specifically  DVI  into  CBT  lessons 
until  video  editing  software  becomes  easier  to  use.  The  danger  with  all  new  technologies  is  that 
placed  in  unskilled  hands  they  may  do  more  harm  than  good.  Without  instructional  design 
expertise  and  some  element  of  creativity,  tools  such  as  multimedia  can  unnecessarily  complicate 
the  instructional  development  process,  obscuring  their  real  purpose  as  ways  to  enhance  well 
designed  instruction  and  make  it  even  more  effective.  Further  development  of  training  aids, 
automated  tools  and  expert  models  should  facilitate  the  integration  of  technology  with 
instructional  design.  Training  aids  for  decision-making,  for  example,  can  provide  more 
comprehensive  and  immediate  feedback  and  assist  users  to  become  aware  of  their  own 
limitations  and  biases  (Wickens  1984). 
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The  goal  of  expert  modeling  is  to  describe  and  simulate  on  a  computer  how  the  expert 
performs  specialized  tasks.  The  behavioral  differences  between  how  the  expert  and  novice 
perform  these  tasks  can  be  identified  and  steps  taken  to  improve  performance.  Models  can 
portray  more  than  just  the  final  outcome  of  a  sequence  of  behaviors;  they  are  capable  of 
representing  the  mechanisms  and  means  of  how  goals  are  attained  (Wickens  1984).  The 
multimedia  development  model  provided  in  this  report  is  a  starting  point  from  which  even  deeper 
behavioral  models  can  be  developed.  From  this  model  tasks,  skills  and  knowledge  requirements 
can  be  determined  for  each  subcomponent  of  multimedia  development  activities.  A  more 
detailed  task  analysis  can  provide  the  basis  for  preparing  novices  to  perform  multimedia 
functions  for  Air  Force  training  organizations  immediately.  In  addition,  when  compared  to  the 
multimedia  behavior  patterns  of  other  experts  the  model  can  be  refined  to  depict  the  affinity  of 
certain  of  its  components  to  automation. 

Future  Use/Validation  of  Observational  Methods 

Informal  methods  of  direct  observation  and  interviews  are  highly  appropriate  for  forming 
initial  hypotheses  for  further  testing.  Interview  data,  for  example,  are  usually  complex,  and 
relatively  unstructured,  but  contain  a  wide  variety  of  information  and  opportunities  for  follow-up 
study.  Techniques  of  knowledge  elicitation  and  cognitive  task  analysis  are  especially  relevant  for 
investigating  relationships  between  people  and  machines,  technology,  and  behavior  (Howell  & 
Cooke  1989).  Future  studies  of  multimedia  development  should  involve  observational  studies  of 
multiple  teams  in  different  development  contexts  for  comparison  and  validation.  Further,  if  two 
or  more  observers  participate,  protocol  analysis  and  inter-rater  studies  can  be  used.  Several  other 
researchers  have  proposed  coding  schemes  which  might  be  an  appropriate  starting  point  for 
future  study  (Goor  and  Sommerfield  1975,  Hamm  1987,  Perez  and  Neiderman  1992).  Feasibility 
studies  utilizing  observational  data  would  be  extremely  useful  for  evaluating  a  new  training 
system  prior  to  implementation. 

While  observational  methods  provide  abundant  data  for  analysis  when  used  in  contexts 
such  as  this,  the  observational  data  is  expensive  in  terms  of  the  resources  required  to  accumulate 
it.  In  addition,  taken  alone  observational  data  cannot  provide  all  the  answers  to  difficult 
questions  about  technology  integration  into  the  instructional  design  process.  Without  coupling 
the  observational  data  with  a  broader  perspective  of  the  entire  Air  Force  instructional  design 
context,  the  data  tends  to  point  to  specific  details  which  do  not  form  patterns  from  which 
behavioral  principles  can  be  extracted. 

Future  Research  on  Multimedia 

Technology  is  evolving  in  the  direction  of  increasing  complexity  and  interrelatedness  of 
elements.  Digital  technology  is  becoming  the  standard  for  communication  and  training.  Audio 
and  video  stored  as  data  can  be  efficiently  and  widely  served  over  networks  (Pea  &  Gomez 
1992).  Computer  manufacturers  may  eventually  incorporate  DVI-like  applications  onto 
motherboards  of  computers  so  that  all  personnel  involved  in  training  (i.e.,  novice  instructional 
designers,  SMEs,  trainers)  will  have  unlimited  access  to  the  technology. 
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The  digital  format  of  multimedia  makes  it  a  versatile  tool  in  presenting  varied 
applications  of  the  same  program  to  users  depending  upon  pre-selected  variables.  For  example, 
digital  multimedia  is  capable  of  delivering  instruction  in  multiple  languages  and  formats 
simultaneously,  which  is  becoming  more  important  for  an  increasingly  multi-cultural  and  multi¬ 
lingual  population  from  which  the  Air  Force  selects  its  trainee  pool  (Kearsley  1991).  This 
potential  to  accommodate  differences  in  learning  aptitudes  and  styles  can  provide  the  expert 
instructional  designer  with  an  additional  tool  to  reach  a  broader  target  population. 

Developments  in  networking  and  telecommunications  will  facilitate  transmitting 
multimedia  data.  The  possibility  of  decentralizing  interactive  training  would  also  provide 
opportunities  to  offer  just-in-time  or  on  demand  access  to  training;  in  this  case,  trainees  could 
access  lessons  exactly  whenever  and  wherever  they  are  needed. 

Potential  Research  Issues 

The  integration  of  multimedia  into  traditional  ISD  poses  some  minor  questions  in  light  of 
recent  modifications  that  the  Air  Force  is  making  in  their  approach  to  that  process.  While  ISD 
was  developed  far  in  advance  of  digital  multimedia  technology,  the  ISD  model  is  still  general 
enough  to  accommodate  such  new  technologies.  Therefore,  the  development  of  principles  and 
guidelines  for  multimedia  should  be  perfectly  compatible  with  ISD.  These  principles  and 
guidelines  can  form  the  basis  for  training  a  cadre  of  Air  Force  instructional  developers  in  the  use 
of  the  technology.  A  reminder  is  in  order  that  multimedia  technology  cannot  replace  sound 
instructional  design.  On  the  other  hand,  when  the  Air  Force  investigates  the  development  of  such 
technologies  as  AIDA  which  deviate  fundamentally  from  traditional  ISD  approaches  to 
instructional  development,  research  will  have  to  be  conducted  as  to  how  newer  technologies  such 
as  multimedia  can  be  integrated  into  these  tools.  The  problem  does  not  appear  to  be  one  of 
compatibility,  rather  it  appears  to  be  finding  out  how  this  integration  can  best  be  achieved. 
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CRITICAL  INCIDENTS 


Ref.  No.  :  1  Date:  03/19/92 

Category:  Technical  Type:  Hardware 

Description:  Decided  not  to  simulate  joystick  action 
Rationale:  Hard  to  replicate  high  fidelity  training  scenario 
Effect.  Students  use  mouse,  focus  on  information  presentation 


Ref.  No.:  2  Date:  03/19/92 

Category:  Technical  Type:  Software 

Description:  Decided  to  work  within  Windows  environment 
Rationale:  Compatibility  with  Armstrong  Lab's  system 
Effect  Restricted  of  software  options  and  capabilities 


Ref.  No.:  3  Date:  05/21/92 

Category:  Technical  Type:  Multimedia 

Description:  Identified  multi-generation/VHS  quality  of  video  (See  No.  9) 
Rationale:  Video  was  at  least  second  generation  copy  of  original 
Effect  Needed  to  decide  if  digitized  image  quality  was  acceptable 


Ref.  No.  :  4  Date:  05/21/92 

Category:  Management  Type:  Design 

Description:  Decided  to  use  modular  lesson  design 

Rationale:  Useful  for  programming,  file  management,  revisions.  Lesson  components  such  as 
"taxiing,"  or  "take  off'  were  designed  to  be  virtually  stand  alone.  Thus  making  it  easier  to 
change  one-out,  or  isolate  to  make  revisions,  or  re-order. 

Effect  Helpful  for  short-term  editing  and  future  updating 


Ref.  No.  :  5  Date:  06/10/92 

Category:  Technical  Type:  Hardware 

Description:  Decided  not  to  use  CD  ROM  for  lesson  delivery 
Rationale:  CD  ROM  is  static,  expensive,  for  single  prototype  system 
Effect  AF  system  would  need  to  have  a  large  hard  drive 


Ref.  No.:  6  Date:  06/10/92 

Category:  Management  Type:  Design 

Description:  Decided  not  to  use  traditional  ISD  approach  to  instructional  design 
Rationale:  Student  characteristics,  learning  outcome  priority,  research  potential 
Effect  Major  contribution  to  instructional  design 
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Ref  No.  :  7  Date:  06/10/92 

Category:  Management  Type:  Design 

Description:  Decision  to  model  the  multimedia  lesson  on  effective  classroom  activities 
Rationale:  Lesson  would  be  more  interesting  and  effective 
Effect.  Multimedia  used  to  support  most  objectives 


Ref.  No.:  8  Date:  06/10/92 

Category:  Management  Type:  Design 

Description:  Developer  analyzed  formation  flying  tasks  and  skills 

Rationale:  Helps  communication  with  SME;  role-plays  student;  effective  to  integrate  flying 

jargon  and  potential  means  of  student  interaction 

Effect.  Personalized  notes,  explanations  become  part  of  training  system 


Ref.  No.  :  9  Date:  07/01/92 

Category:  Management  Type:  Design 

Description:  Decision  to  use  parts  of  existing  T-37  video  footage 
Rationale:  Contained  adequate  information  for  multimedia  elements 
Effect.  Training  video  would  have  decreased  quality,  fidelity 


Ref.  No.:  10  Date:  07/10/92 

Category:  Management  Type:  Scheduling 

Description:  Decided  not  to  talk  with  student  pilots  at  Laughlin  AFB 
Rationale:  Hard  to  schedule;  too  time  consuming  and  expensive 

Effect.  No  new  original  video  or  direct  input  from  student  pilots  needed  or  required  for 
successful  training  implementation 


Ref.  No.:  11  Date:  07/10/92 

Category:  Management  Type:  Design 

Description:  Planned  use  of  innovative  interface  techniques  in  system,  i.e.,  attention  loop  to 
motivate,  extensive  assessment,  text-based  coaching  and  remediation. 

Rationale:  Increase  student  motivation  and  use  of  system 
Effect:  Added  variety/opportunities  to  interact  with  lesson 


Ref.  No.:  12  Date:  07/10/92 

Category:  Management  Type:  Design 

Description:  Emphasized  critical  visual  references  students  need  to  know  to  fly  in  formation 
Rationale:  Crucial  concept  to  acquire  formation  flying  skills 
Effect.  Utilize  DVI  to  best  advantage  for  illustrating  concepts 
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Ref.  No.  :  13  Date :  07/10/92 

Category:  Management  Type:  Design 

Description:  Identify  help  and  remediation  elements  for  development 
Rationale:  Modularity  of  system  could  support  more  lesson  elements  in  future 
Effect.  Potential  to  support  embedded  expert  system 


Ref.  No.  :  14  Date:  07/16/92 

Category:  Management  Type:  Design 

Description:  Decided  to  combine  and  limit  number  of  lesson  modules 
Rationale:  Developer  planned  to  do  too  much  originally 
Effect.  Modules  divided  into  sections  and  separate  books 


Ref.  No.:  15  Date:  07/16/92 

Category:  Technical  Type:  Multimedia 

Description:  Decided  not  to  fly  sortie  to  obtain  new  T-37  video  footage 
Rationale:  Too  expensive  and  time  consuming  to  justify 
Effect.  Use  existing  2nd+  generation  video;  old  paint  scheme 


Ref.  No.  :  16  Date:  07/16/92 

Category:  Management  Type:  Design 

Description:  Decided  to  focus  on  fingertip  formation  and  rejoins 
Rationale:  Most  effectively  shows  off  DVI  capabilities 
Effect.  More  intensive  effort  for  fewer,  more  robust  modules 


Ref.  No.  :  17  Date:  08/07/92 

Category:  Technical  Type:  Hardware 

Description:  Instead  of  buying  turnkey  system,  configured  system 
Rationale:  More  powerful  system;  lower  cost;  learning  experience 
Effect.  Takes  longer  to  get  up  and  running;  provides  more  versatile  system 


Ref.  No.:  18  Date:  08/07/92 

Category:  Technical  Type:  Software 

Description:  Decided  to  purchase  Toolbook  instead  of  Authorware 
Rationale:  Mostly  because  of  prohibitive  cost,  AF  preference 
Effect.  Takes  more  time  to  learn  how  to  use;  harder  to  debug 
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Ref.  No.:  19  Date :  08/07/92 

Category:  Management  Type :  Design 

Description :  Constructed  learning  objectives  document 
Rationale :  ATC  needs  well  defined  statements  of  learning 
Ejgfccf:  Confirmed  definitions  and  prioritization  of  objectives 


Kef.  No.:  20  Date:  08/12/92 

Category:  Management  Type:  Interpersonal 

Description:  Needed  to  produce  separate  learning  objectives  document 

Rationale:  Originally  expected  SME  to  edit  other  documents  to  produce  this  information 

Effect.  Developer  adjusted  his  design  expectations 


Ref.  No.:  21  Date:  08/20/92 

Category:  Management  Type:  Design 

Description:  SME  stressed  important  "air  discipline,"  in  a  narrow  flying  environment 
Rationale:  Training  required  specialization  and  demonstration 
Effect.  Focused  on  checklists  and  commands,  not  formulating  concepts 


Ref.  No.:  22  Date:  08/21/92 

Category:  Management  Type:  Interpersonal 

Description:  Developer  based  discussion  on  unique  examples  or  claims  about  topic  to  which 

SME  reacted  by  agreeing  or  making  substantial  changes 

Rationale:  Helped  developer  find  out  what  SME  really  wanted 

Effect.  Demonstrated  that  this  was  an  effective  technique  for  generating  discussion 


Ref.  No.:  23  Date:  08/21/92 

Category:  Technical  Type:  Multimedia 

Description:  Decision  to  use  Steve  Young  to  narrate  script 
Rationale:  A  clear  and  deep  voice  is  best  for  digitizing 
Effect  Produced  quality,  "space  saving"  narration 


Ref.  No.  :  24  Date:  08/21/92 

Category:  Technical  Type:  Software 

Description:  Realized  constraints  of  working  with  Beta  version 
Rationale:  Only  cost  effective  and  Windows  software  available  at  the  time 
Effect.  Resulted  in  a  slower  learning  curve;  less  versatile  system 
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Ref.  No.:  25  Date:  08/21/92 

Category:  Management  Type:  Design 

Description:  Multimedia  developer  claims  potential  conflict  between  ISD  model  and  multimedia 
approach 

Rationale:  ISD  not  intended  to  account  for  all  creative  elements 
Effect.  Cannot  formfit  creativity  into  ISD  model 


Ref.  No.:  26  Date:  09/04/92 

Category:  Management  Type:  Scheduling 

Description:  Unplanned  delays  in  hardware/software  availability 
Rationale:  Vendors  unable  to  release  products  on  announced  schedule 
Effect  Produced  a  delay,  but  allowed  more  time  for  analysis/design 


Ref  No.:  27  Date:  09/09/92 

Category:  Management  Type:  Design 

Description:  Constructed/distributed  instructor  pilot  survey 

Rationale:  Avoided  travel  costs;  sampled  more  than  one  base 

Effect  Provided  data  on  personal  experiences  and  expressions  used  during  instruction 


Ref  No.:  28  Date:  09/17/92 

Category:  Management  Type:  Design 

Description:  Developed  plan  to  combine  aspects  of  system/student-driven  navigation 
Rationale:  Student  must  master  basic  knowledge  to  continue  lesson 
Effect  Innovative  way  of  assessing  information  mastered 


Ref.  No.:  29  Date:  09/22/92 

Category:  Management  Type:  Design 

Description:  Used  audio  narration  script  as  main  knowledge  piece 
Rationale:  Way  of  organizing  universe  of  knowledge 
Effect.  Selected  learning  objectives  from  "universe" 


Ref.  No.:  30  Date:  09/22/92 

Category:  Management  Type:  Design 

Description:  Instructor-pilot  survey  was  reworked  and  depersonalized 
Rationale:  Test  developer  made  it  psychometrically  correct 
Effect  Reworded  survey  de-emphasized  eliciting  personal  expressions 
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Ref.  No.  :  31  Date:  10/09/92 

Category:  Technical  Type:  Software 

Description:  Created  software  program  to  lock  out  mouse  interrupts 
Rationale:  Reduce  confusion  about  system  hanging  up 
Effect.  Students  not  distracted  by  inadvertent  selections 


Ref.  No.  :  32  Date:  10/09/92 

Category:  Management  Type:  Design 

Description:  Specified  which  media  accompany  narration 
Rationale:  Developer  planned  to  solicit  support  information  from  SME 
Effect  Developer  selected  media  to  support  learning  strategies 

Ref.  No.  :  33  Date:  10/09/92 

Category:  Management  Type:  Design 

Description:  Discussion  of  student  vs.  system-driven  navigation 
Rationale:  Continuing  debate  over  how  student  should  access  information 
Effect.  Decided  on  overall  system  directed  navigation,  with  some  student  input 

Ref.  No.:  34  Date:  10/09/92 

Category:  Technical  Type:  Multimedia 

Description:  Demonstrated  limitations  in  DVI  capture  routine 

Rationale:  Could  display  infinite  color  resolution  with  Targa;  current  display  uses  256  colors 
Effect.  Purchased  high  color  VGA  board  to  improve  image 

Ref.  No.:  35  Date:  10/09/92 

Category:  ^Management  Type:  Design 

Description:  Discussion  of  auditory  vs.  text  learning 
Rationale:  SME  emphasized  text;  developer  wanted  parallel  information 
Effect  Developer  would  produce  balanced  media  presentations 


Ref.  No.:  36  Date:  10/09/92 

Category:  Management  Type:  Design 

Description:  Prototyped  lesson  on  screen,  instead  of  storyboards 

Rationale:  Developer  working  alone  had  ideas  in  his  head 

Effect  SME  wanted  to  see  comparisons  between  previous  and  current  versions 
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Ref.  No.:  37  Date :  10/29/92 

Category:  Management  Type :  Design 

Description:  SME  was  not  enthusiastic  about  interface  concepts 

Rationale:  SMEs  had  option  to  suggest  creative  ways  of  demonstrating  concepts 

Effect.  SME  expressed  no  visible  preference  at  this  stage  of  development 


Ref.  No.:  38  Date:  11/02/92 

Category:  Technical  Type:  Multimedia 

Description:  Animation  was  more  difficult  and  took  longer  than  expected 
Rationale:  Problem  with  availability  of  software  and  documentation 
Effect,  used  restricted  number  of  custom  animation  segments 


Ref.  No.:  39  Date:  11/04/92 

Category:  Management  Type:  Design 

Description:  Different  narrators  selected  for  content  sections 
Rationale:  Helped  distinguish  between  basic/more/test  information 
Effect:  Consistent  presentation  of  information  to  users 


Ref.  No.  :  40  Date:  11/04/92 

Category:  Management  Type:  Design 

Description:  Developer  "coached"  narrators  during  audio  taping 
Rationale:  Anticipated  how  audio  will  be  used  by  learners 
Effect:  Coaching  produced  audio  required  for  effective  teaching 


Ref  No.  :  41  Date:  11/04/92 

Category:  Technical  Type:  Multimedia 

Description:  DB  levels/subject's  distance  from  microphone  controlled 
Rationale:  Inconsistent  volume  level  produced  distracting  audio 
Effect:  Developer  coached  narrators  for  correct  enunciation 


Ref.  No.  :  M  Date:  11/12/92 

Category:  Management  Type:  Design 

Description:  Produced  audio  narration  "edit  decision  list" 

Rationale:  Tool  was  used  to  create  template;  combined  multimedia  elements 
Effect:  Used  as  storyboard  script  and  "universe  of  knowledge" 
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Ref.  No.:  43  Date:  11/12/92 

Category:  Technical  Type:  Software 

Description :  Windows  DVI  software  lost  some  editing  capabilities 
Rationale:  Beta  version  included  only  capture,  playback  tools 
Effect.  Decided  to  wait  for  software  or  go  to  production  house 


Ref  No.  :  44  Date:  11/12/92 

Category :  Technical  .  Type:  Multimedia 

Description:  Development  team  was  unable  to  capture  still  image 
Rationale:  Must  retrieve  and  play  video  file  each  time 
Effect.  Took  a  lot  of  time  to  access  files 


Ref.  No.  :  45  Date:  11/16/92 

Category:  Technical  Type:  Multimedia 

Description .  Tested  different  data  storage  and  retrieval  formats 
Rationale :  Attempt  to  find  quickest  way  to  open  and  close  files 
Effect.  Produced  better  coordination  of  multimedia  elements 


Ref  No.  :  46  Date:  11/16/92 

Category:  Technical  Type:  Software 

Description:  Organized  lesson  elements  into  two  big  AVS  files 
Rationale:  Avoided  delay  in  accessing  information 
Effect.  Eased  file  management;  less  time  required  to  open  files 


Ref.  No.  :  47  Date:  11/18/92 

Category:  'Technical  Type:  Software 

Description:  Little  or  no  documentation  for  Windows  DVI  software 

Rationale:  Beta  version  lacked  documentation;  poor  technical  support 

Effect.  Team  developed  communications  network  to  exchange  product  information 


Ref.  No.:  48  Date:  11/18/92 

Category:  Technical  Type:  Multimedia 

Description:  Selected  post-production  video  and  digitization 
Rationale:  Windows  DVI  editing  software  not  yet  available 
Effect.  Unable  to  develop,  utilize  in-house  edit  routines 
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Ref  No.  :  49  Date:  11/18/92 

Category:  Technical  Type:  Multimedia 

Description:  Audio  clips  matched  with  video  clips  cannot  change 
Rationale:  Did  not  have  software  algorithm  to  combine  files 
Effect  Unable  to  reuse  data;  lost  flexibility  for  system 


Ref  No.  :  50  Date:  11/18/92 

Category:  Technical  Type:  Multimedia 

Description:  Constructed  graphics  elements  list  with  SME  input 
Rationale:  Time  to  organize  and  name  files  for  production 
Effect  Reviewed  video  footage  and  captured  still  images 


Ref  No.  :  51  Date:  11/24/92 

Category:  Technical  Type:  Multimedia 

Description:  Scripting  arranged  to  reuse  some  routines 
Rationale:  Routines  were  repeated  through  sections  of  book 
Effect  Reduced  amount  of  time  spent  on  new  scripting 


Ref.  No.  :  52  Date:  12/01/92 

Category:  Technical  Type :  Multimedia 

Description:  Standardized  transitions  between  video  clips 
Rationale:  Reduced  potential  distraction  to  learning 
Effect  Limited  set  of  transitions  selected  for  video  clips 


Ref.  No.:  53  Date:  12/01/92 

Category:  Management  Type:  Design 

Description:  Videotaped  SMEs  demonstrations  of  training  concepts 
Rationale:  Obtained  original,  and  real  instructional  footage 
Effect  Provided  customized  training  system  details 


Ref.  No.:  54  Date:  12/01/92 

Category:  Technical  Type:  Multimedia 

Description:  Frame  counters  were  different  for  audio  and  video  clips 
Rationale:  Different  frame  counters  were  based  on  time/length  segments 
Effect:  Caused  significant  delays  in  editing  process 
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Ref.  No.  :  55  Date  :  12/01/92 

Category:  Management  Type:  Design 

Description:  SMEs  pointed  out  content  inconsistency 
Rationale:  Used  SMEs  to  review  content  integrity 
Effect.  Improved  quality  of  lesson  with  existing  material 


Ref.  No.:  56  Date:  12/01/92 

Category:  Management  Type:  Design 

Description:  Developer  encouraged  good  video  technique  for  SMEs 
Rationale:  Considered  videotape  as  major  multimedia  element 
Effect.  Obtained  good  video  footage  of  SMEs  demonstrations 


Ref  No.  :  57  Date:  12/03/92 

Category:  Technical  Type:  Multimedia 

Description:  Decide  to  use  RTV  instead  of  PLV 
Rationale:  Less  time  consuming  and  expensive  to  produce 
Effect.  Images  captured  with  limited  fidelity 


Ref.  No.:  58  Date:  12/03/92 

Category:  Technical  Type :  Multimedia 

Description:  Experimented  with  image  quality  in  DVI  capture  machine 
Rationale:  Different  wiring  might  have  affected  image  quality 
Effect.  Professional  videotape  player  produced  best  image 


Ref.  No.:  59  Date:  12/04/92 

Category:  Technical  Type:  Software 

Description:  Could  key  on  frame  number  but  not  audio  text 
Rationale:  No  MCI  command  existed  for  this  capability 
Effect.  Could  not  overlay  graphics  in  real  time;  keyed  on  audio 


Ref.  No.:  60  Date:  12/04/92 

Category:  Technical  Type:  Design 

Description:  Problem  noticed  with  CGA  resolution  display  of  Targa  image 
Rationale:  There  was  not  enough  room  for  information  to  display  high  quality  image 
Effect.  Compatible  driver  purchased  to  display  full  screen  images 
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Ref  No.  :  61  Date:  12/04/92 

Category:  Technical  Type:  Design 

Description:  Problem  noticed  with  maintaining  detail  when  shrinking  images 
Rationale:  Data  loss  was  relative  to  the  amount  image  was  scaled  down 
Effect:  Used  visual  judgment  to  scale  images 


Ref  No.:  62  Date:  12/07/92 

Category:  Technical  Type:  Multimedia 

Description:  Audio  narration  required  more  takes  than  expected 
Rationale:  There  were  analog  to  digital  conversion  glitches 
Effect:  Process  took  longer  and  required  more  patience  for  re-takes 


Ref.  No.  :  63  Date:  12/01/92 

Category:  Technical  Type:  Design 

Description:  Created  table  for  origins  and  destinations  of  multimedia  elements 

Rationale:  Facilitated  programming  of  playback  frames 

Effect.  Facilitated  working  on  lesson  navigation  system 


Ref.  No.:  64  Date:  12/07/92 

Category:  Technical  Type:  Hardware 

Description:  Discussion  about  upgrading  Targa  board 
Rationale:  Would  permit  greater  variety  of  special  effects 

Effect.  Decided  against  upgrade  because  it  was  too  expensive  and  of  limited  benefit 


Ref.  No.:  65  Date:  12/07/92 

Category:  Technical  Type:  Multimedia 

Description :  Maintained  only  one  version  of  audio  clip  on  tape 
Rationale:  Reduced  possibility  of  digitizing  or  capturing  wrong  file 
Effect.  Produced  one  correct  version  of  audio  tape  narration 


Ref  No.:  66  Date:  12/09/92 

Category:  Technical  Type:  Software 

Description:  Network  installed  to  link  the  two  DVI  development  computers  together 

Rationale:  Facilitated  dumping  files  and  sharing  software 

Effect.  Allowed  developers  to  work  on  files  simultaneously 
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Ref.  No.  :  67  Date :  12/09/92 

Category :  Technical  Type :  Multimedia 

Description:  Noted  significant  deviations  in  digitized  audio  file 
Rationale:  Inconsistent  audio  level  noted  during  mastering  and  production 
Effect.  Required  manually  redigitizing  master 


Ref.  No.:  68  Date:  12/11/92 

Category:  Technical  Type:  Hardware 

Description:  New  AF  DVI  development  system  did  not  work  correctly 
Rationale:  System  vendor  did  not  correctly  configure  system 
Effect:  AF  system  not  compatible  with  development  team's  computer 


Ref  No.  :  69  Date:  12/11/92 

Category:  Technical  Type:  Software 

Description:  Found  that  Toolbook  can  handle  FLC  files 
Rationale:  Could  not  animate  very  well  in  Toolbook 
Effect.  Discovered  tool  for  implementing  animated  graphics 


Ref.  No.  :  70  Date:  12/11/92 

Category:  Technical  Type:  Multimedia 

Description:  Decided  how  to  record  music  for  training  system 

Rationale:  Alternatives  were  to  use  DVI  or  MIDI  with  multiple  tracks  (non  DVI) 

Effect.  Digitized  live  audio  with  AMII  (required  no  specific  boards) 


Ref.  No.:  71  Date:  12/19/92 

Category:  Technical  Type:  Multimedia 

Description:  Bleeding  noticed  around  margins  of  DVI  window  border 
Rationale:  Initially  used  poor  quality  video  equipment 
Effect:  Overlaid  border  with  margin  of  video  lines 


Ref.  No.:  72  Date:  12/22/92 

Category:  Technical  Type:  Multimedia 

Description:  Development  team  used  point-to-point  editing 
Rationale:  Used  old  editing  software  with  new  DVI  boards 
Effect.  This  was  an  exceptionally  time  consuming  way  of  editing 
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Ref.  No.:  73  Date:  01/04/93 

Category:  Technical  Type:  Hardware 

Description:  Mei’s  and  AFs  system/monitor  cabling  configurations  were  different 
Rationale:  Converted  AFs  two  monitor  system  to  single  monitor  system 
Effect:  VESA  connector  supplied  new  video  features 


Ref  No.:  74  Date:  01/11/93 

Category:  Technical  Type:  Multimedia 

Description:  Established  formal  video  log  for  ATC  raw  footage 
Rationale:  Needed  to  organize  information  about  frame,  location,  time 
Effect:  Facilitated  obtaining  video  files  for  production  and  editing 


Ref.  No.:  75  Date:  01/19/93 

Category:  Technical  Type:  Software 

Description:  Book  does  not  run  on  AFs  system 
Rationale:  Computer  does  not  recognize  the  MCI  command 
Effect:  To  correct  the  problem  MCI  drivers  were  removed  and  reinstalled 


Ref.  No.  :  76  Date:  01/19/93 

Category:  Technical  Type:  Software 

Description:  Digitized  video  appeared  jumpy,  dropping  frames  during  capture 
Rationale:  VERIFY  ON  in  AUTOEXEC.BAT  file  required  too  much  time 
Effect.  Turning  off  command  resolved  problem 

Ref  No.  :  77  Date:  01/20/93 

Category:  Technical  Type:  Software 

Description:  After  entire  file  plays,  image  no  longer  fades  to  black 
Rationale:  Image  remained  after  file  closed;  caused  time  delay  to  correct 
Effect.  A  VS  file/30  frames  black  video  appended  to  files 


Ref  No.:  78  Date:  01/21/93 

Category:  Technical  Type:  Software 

Description:  Progress  bar  did  not  accurately  show  remaining  part  of  lesson 
Rationale:  Repeated  tracks  counted  towards  progress  in  lesson 
Effect:  Progress  bar  revised  to  track  only  new  information 
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Ref.  No.:  79  Date:  01/21/93 

Category:  Technical  Type:  Software 

Description:  Decision  made  about  displaying  cursor  and  associated  audio 
Rationale:  Cursor  would  indicate  for  student  to  wait;  the  selection  was  unavailable 
Effect.  Student  sees/hears  obvious  cues  of  system's  status 


Ref.  No.:  80  Date :  01/21/93 

Category:  Technical  Type:  Multimedia 

Description:  Problem  noted  of  locating/ordering  clips  in  different  files 
Rationale:  Initially  used  prototype  approach;  later,  used  generic  code 
Effect:  Team  modified  clip  "look  up  table" 


Ref.  No.:  81  Date:  01/21/93 

Category:  Technical  Type:  Hardware 

Description:  Amplified  speakers  were  not  responding  to  some  sounds 
Rationale:  Built-in  power  down  circuit  required  time  to  receive  signal 
Effect.  Modified  speaker  power  down  circuit 
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APPENDIX  B: 

MULTIMEDIA  JOB/TASK  ANALYSIS 
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Multimedia  Job  and  Task  Analysis 


1.0  Multimedia  Development 

1.1  Audio 

1.1.1  Integrate  audio  capabilities  of  hardware  and  software 

1.1.2  Select  audio  elements 

1.1.3  Define  the  learning  elements 

1.1.4  Develop  scripts  for  each  learning  element 

1.1.5  Identify  common  student  mistakes 

1.1.6  Define  and  apply  alternative  explanation  methods 

1.1.7  Develop  script  for  narration 

1.1. 7.1  Develop  scripts  for  positive  feedback,  remediation,  and  help 

1.1. 7.2  Develop  scripts  for  negative  feedback,  remediation,  and  help 

1.1. 7.3  Ensure  correctness  of  script  content,  language,  pronunciation 

1.1. 7 .4  Define  areas  of  inflection,  confidence  and  knowledge 

1. 1.7.5  Identify  where  to  use  male  and  female  voice  talent 

1.2  Sound  Effects/Music 

1 .2. 1  Define  special  areas  for  effects  to  support 

1.2.2  Develop  script  for  effects 

1.2.3  Ensure  correctness,  appropriateness,  legal  copyrights,  etc. 

1.2.4  Select,  and  test  effects  in  application 

1.3  Graphic/ Animation 

1.3.1  Define  graphic  list 

1.3.2  Work  from  storyboard  to  develop  list 

1.3.3  Use  video  images  if  possible 

1.3.4  Determine  file  size,  load  time,  etc. 

1.3*5  Use  graphics  package  to  create  and  edit  graphics 

1.3.6  Define  animations 

1.3.7  Define  screen  designs 

1.3.7. 1  Pay  attention  to  consistency  of  appearance  and  operation 

1. 3.7.2  Define  color  standards 

1. 3.7.3  Ensure  clarity  of  instructional  objective 

1.4  Still  Image  (Video,  Photo) 

1.4.1  Identify  images  that  can  be  obtained  via  video  capture 

1.4.2  Use  full  motion  video  plan 

1.4.3  Create  a  shot  sheet  for  stills 

1.4.4  Work  stills  shot  sheet  into  full  motion  video  shot  sheet 

1.4.5  Use  proper  video  formats  for  graphics  software  and  application  compatibility 

1.4.6  Establish  still  image  log  or  image  record  to  facilitate  development 


52 


1.5  Motion  Video 

1 .5. 1  Identify  images  that  can  be  obtained  via  video  capture 

1.5.2  Create  a  shot  sheet  for  full  motion 

1.6  Authoring/Software 

1.6.1  Develop  prototype 

1.6.2  Design  system  help 

1.6.3  Storyboard  navigation  through  system 

1.6.4  Define  instructional  activities  and  goals 

1.6.5  Develop  overview  of  course 

1.6.6  Identify  lesson  composition  and  organization 

1.6.7  Identify  subordinate  teaching  topics 

1.6.8  Storyboard  user  interface 

1.6.9  Test  screen  designs  with  instructional  needs 

2.0  Multimedia  Production 

2.1  Video  Production 

2.1.1  Work  from  shot  sheet  (know  image  and  stills  requirements  also) 

2.1.2  Cross-check  the  design  of  the  shot  sheet 

2. 1 .2. 1  Check  that  all  conditions  accounted  for 

2. 1 .2.2  Edit  the  shot  sheet  to  reflect  changes 

2.1.3  Ensure  shot  sheet  reflects  time  and  cost  concerns 

2.1.4  Determine  shoot  location 

2. 1.4.1  Studio  shoot 

2.1.4.1.1  Prepare  equipment 

2.1.4.1.2  Prepare  the  set 

2. 1.4. 1.3  Check  lighting 

2. 1.4. 1.4  Practice  shot  under  various  conditions  for  best  results 

2. 1.4. 1.5  Determine  if  going  to  video  or  capturing 

2. 1.4. 1.5.1  Determine  special  conditions  for  capturing 

2. 1.4. 1.5. 2  Determine  special  conditions  for  tape 

2. 1.4. 1.6  Record  progress  and  helpful  notes  into  video  log 

2. 1.4.2  Location  shoot 

2. 1.4.2. 1  Determine  transportation  and  its  ramifications 

2. 1.4. 1.2  Prepare  equipment 

2. 1.4. 1.3  Understand  natural  light,  weather,  local  authorities  and  legal  issues, 
Permits  etc. 

2. 1.4. 1.4  Practice  set-up  and  breakdown  for  various  shots  at  a  location 

2. 1.4. 1.5  Plan,  practice  and  prepare  for  one-time  chance  at  getting  the  right  shot 

2. 1.4. 1.6  Determine  if  going  to  video  or  capturing 

2. 1 .4. 1 .6. 1  Determine  special  conditions  for  capturing 

2. 1 .4. 1 .6.2  Determine  special  conditions  for  tape 

2. 1.4. 1.7  Record  progress  and  helpful  notes  into  video  log 

2.1.5  Prepare,  create,  acquire  application  related  materials,  props,  etc. 

2.1.6  Select  Camera  Type  and  Tape  Stock 


2.1.7  Prepare,  coach,  instruct  etc.,  the  talent 

2.1.8  Practice  run  through  the  shoot 

2.1.9  Identify  and  fix  problems 

2.1.10  Shoot  the  video 

2.2  Still  Image  Production 

2.2.1  Work  from  shot  sheet 

2.2.2  Cross-check  the  design  of  the  shot  sheet  to  include  still  image  requirements 
obtained  From  camera  or  video 

2.2.3  Check  that  all  still  images  and  conditions  accounted  for 

2.2.3. 1  Edit  the  shot  sheet  to  reflect  change 

2.23.2  Ensure  shot  sheet  reflects  time  and  cost  concerns 

2.2.4  Determine  shoot  location 

2.2.4. 1  Studio  shoot 

2.2.4. 1.1  Prepare  equipment 

2.2.4. 1.2  Prepare  the  set 

2.2.4. 1.3  Check  the  lighting 

2.2.4. 1.4  Practice  shot  under  various  conditions  for  best  results 

2.2.4. 1.5  Determine  if  going  to  video  or  capturing 

2.2.4. 1.5.1  Determine  special  conditions  for  capturing 

2.2.4. 1.5.2  Determine  special  conditions  for  tape 

2.2.4. 1.6  Record  progress  and  helpful  notes  into  video  log 
2.2A.2  Location  shoot 

2.2.4.2.1  Determine  transportation  requirements  and  its  ramifications 

2.2.4.2.2  Prepare  equipment 

2.2.4.2.3  Understand  natural  light,  weather,  local  authorities  and  legal  issues, 
permits  etc. 

2.2.4.2.4  Practice  set-up  and  breakdown  for  various  shots  at  a  location 

2.2.4.2.5  Plan,  practice  and  prepare  for  a  one-time  chance  at  getting  the  right  shot 

2.2.4.2.6  Determine  if  going  to  video  or  capturing 

2.2.4.2.6.1  Determine  special  conditions  for  capturing 

2.2.4.2.6.2  Determine  special  conditions  for  tape 

2.2.4.2.7  Record  progress  and  helpful  notes  into  video/image  log 

2.2.5  Determine,  prepare  resources 

2.2.5. 1  Determine  lighting  requirements 

2.2.5.2  Prepare,  create,  acquire  application  related  materials,  props,  etc. 

2.2.5.3  Select  camera  type  and  tape  stock 

2.2.6  Prepare,  coach,  instruct  etc.,  the  talent 

2.2.7  Determine  capturing  (digitizing)  requirements 

2.2.8  Practice  run  through  the  shoot 

2.2.9  Identify  and  fix  problems 

2.2.10  Shoot  the  image 
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2.3  Graphic 

2.3.1  Ensure  the  design  of  the  graphic  list  accounts  for  the  application's  graphics 
requirements 

2.3.2  Create  graphic  elements 

2.3.3  Ensure  graphics  software  is  capable  of  various  formats 

2.3.4  Create  graphics  (and  images)  in  a  single  color  palette 

2.3.5  Use  database  and  helpful  notes  to  log  the  completed  graphics 

2.4  Animation 

2.4. 1  Work  from  storyboards,  graphic  lists,  etc. 

2.4.2  Determine  type  and  level  of  fidelity  of  animation  (3-D,  stick  figure,  etc.) 

2.4.3  Ensure  the  (animation)  software  format  supports  application  format 

2.4.4  Create  animation 

2.4.5  Record  animation  data  in  database  use  helpful  notes 

2.5  Audio 

2.5.1  Work  from  storyboards,  shot  sheets  and  other  development  tools  that  define  audio 
requirements 

2.5.2  Identify  recording  needs 

2.5.3  Prepare  equipment 

2.5.4  Prepare,  coach,  etc.  talent 

2.5.5  Record  audio 

2.5.6  Re-record  if  necessary 

2.5.7  Use  log  to  record  audio  information 

2.6  Music 

2.6.1  Work  from  storyboards,  shot  sheets  and  other  development  tools  that  define  the 
music  requirements  of  the  application 

2.6.2  Identify  recording  needs 

2.6.3  Prepare  equipment 

2.6.4  Prepare,  coach,  etc.  talent 

2.6.5  Identify  Royalty  Requirements 

2.6.6  Record  music 

2.6.7  Re-record  music  if  necessary 

2.6.8  Use  log  to  record  music  information 

2.7  Sound  effects 

2.7.1  Work  from  storyboards,  shot  sheets  and  other  development  tools  that  define  sound 
effects  requirements 

2.7.2  Identify  recording  needs 

2.7.3  Prepare  equipment 

2.7.4  Prepare,  coach,  etc.  talent 

2.7.5  Record  effects 

2.7.6  Re-record  if  necessary 

2.7.7  Use  log  t  record  effects  information 
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2.8  Graphic  user  interface 

2.8.1  Work  from  the  storyboards,  graphic  lists,  etc. 

2.8.2  Ensure  the  design  of  the  interface  supports  the  application's  instructional 
requirements 

2.8.3  Create  graphic  elements  of  the  interface 

2.8.4  Ensure  graphics  software  is  compatible  with  application  (authoring)  software 

2.8.5  Create  graphics  (and  images)  in  a  single  color  palette 

2.8.6  Use  database  and  helpful  notes  to  log  the  completed  graphics 

2.9  Edit 

2.9.1  Select  element  to  edit 

2.9.2  Motion  video  edits 

2.9.2. 1  Use  highest  quality  sources 

2.9.2.2  Create  edit  decision  list 

2.9 .2.3  Use  on-line  (digital)  or  off-line  (analog)  editing 

2.9.2.4  Edit  video 

2.9.2.5  Record  into  video  log 

2.9.3  Still  image  edits 

2.9.3. 1  Use  highest  quality  sources 

2.9.3.2  Create  image  edit  list 

2.9.3.3  Recapturing  may  be  necessary  for  image  editing 

2.9.3.4  Edit  image 

2.9.3.5  Record  into  image/video  log 

2.9.4  Graphic  edits 

2.9.5  Animation  edits 

2.9.6  Authoring/programming  edits 

2.9.7  Instructional  integrity  edits 

2.9.8  Sound  (music,  audio,  effects)  edits 

2.9.9  Application  edits 

2.9.9. 1  Test  application  against  design  specifications 

2.9.9.2  Note  problems  and  develop  a  corrective  plan 

2.9. 10  Ensure  edits  meet  customer  requirements 

3.0  Authoring/Programming 

3.1  Organize  production  components 

3.1.1  Ensure  each  component  is  complete 

3.1.2  Create,  redo,  or  add  to  a  component 

3.2  Integrate  components  into  application 

3.2.1  Use  guidelines  set  by  interactive  and  software  design 

3.2.2  Work  off  the  storyboards 

3.2.3  Work  off  the  shot  sheets 

3.2.4  Work  off  the  graphic/animation  list 

3.2.5  Work  off  SME's  suggestions 

3.2.6  Build  on  the  prototype 
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3.2.7  Work  with  instructional  designer 

3.3  Use  authoring  system  to  program  components  into  the  system 

3.3.1  Test  and  refine  code 

3.3.2  Document  lessons  learned 

3.3.3  Decide  if  previously  completed  software  or  hardware  could  be  cost-effectively 
improved  by  implementing  the  lessons  learned 
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APPENDIX  C: 

MULTIMEDIA  DEVELOPMENT  MODEL 
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Multimedia  Development  Model 


Appendix  C  is  a  model  of  multimedia  development.  The  format  used  throughout  this 
appendix  is  top-down  structured  analysis.  This  methodology  consists  of  three  separate  but 
integrally  linked  components:  1)  data  flow  diagrams  (graphic  depictions  of  the  system's 
functions  and  relationships),  2)  process  descriptions  (a  specification  or  description  of  each 
function  depicted  in  the  data  flow  diagrams),  and;  3)  data  dictionary  (explanation  of  each  term 
used  in  the  data  flow  diagrams  and  process  descriptions).  The  following  is  a  brief  outline  of  the 
methodology  used  throughout  this  appendix. 

Data  Flow  Diagrams 

These  functional  diagrams  of  the  system  are  organized  in  a  hierarchical  fashion  to  provide 
the  level  of  detail  required  by  the  reader  to  understand  the  system;  how  it  works  and  what  it  does. 
Lower,  and  more  detailed  functional  diagrams  are  often  used  by  systems  analysts  and  designers 
to  develop  detailed  specifications  for  automated  systems.  The  data  flow  diagrams  depict 
graphically  the  internal  processes  which  take  place  in  a  system,  as  well  as  the  relationships  that 
the  system  being  described  has  with  any  other  external  systems.  The  relationship  of  the  functions 
or  processes  which  are  depicted  in  the  data  flow  diagrams  can  be  shown  in  a  hierarchy  structure 
chart,  such  as  the  one  preceding  the  data  flow  diagrams  for  the  multimedia  development  model. 
This  hierarchy  structure  chart  shows  how  each  function  explodes  into  a  more  detailed  description 
on  a  lower  level  data  flow  diagram.  Each  item  listed  on  the  hierarchy  chart  consists  of  a  separate 
data  flow  diagram  and  associated  process  descriptions. 

Several  special  symbols  are  used  in  the  data  flow  diagrams.  Each  symbol  has  a  specific 
meaning  as  follows: 

Represents  information  or  a  data  flow. 

Represents  a  process  or  function. 


Represents  an  information  or  data  file. 


Stands  for  a  system  external  to  the  one  described. 


o 

Data  Store 


Process  Descriptions 

Each  process,  function  or  activity  which  is  depicted  in  a  data  flow  diagram  has  a  process 
description  associated  with  it.  The  process  description  describes  how  the  process  occurs,  in  other 
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words  the  rules  governing  the  operation  or  behavior  of  the  process.  Process  descriptions  tell  how 
an  input  or  some  information  is  used  or  transformed  by  the  process,  or  how  new  information  is 
generated  by  the  process.  Process  descriptions  for  higher  level  functions  may  explode  into  lower 
level  data  flow  diagrams  which  are,  in  turn,  explained  in  more  detail  by  the  process  descriptions 
of  these  sub-processes. 

Data  Dictionary 

Each  item  depicted  in  the  data  flow  diagrams  or  used  in  the  process  descriptions  is 
described  or  defined  in  the  data  dictionary.  The  data  dictionary  is  used  to  describe  the  data  flows, 
data  stores  and  external  systems.  Note  that  there  is  no  data  dictionary  entry  corresponding  to  a 
process.  Processes  or  functions  are  explained  fully  by  process  descriptions. 

•  Data  Flow.  These  are  represented  on  the  data  flow  diagrams  by  an  arrow.  The  point  of 
the  arrow  indicates  the  direction  in  which  the  information,  report  or  data  flows.  The  label 
is  the  name  of  the  information  or  data. 

•  Data  Store.  These  items  are  represented  in  the  data  flow  diagrams  by  a  set  of  parallel 
lines.  The  label  is  the  common  name  for  the  store.  The  data  stores  represent  files  which 
are  maintained  by  the  system  being  described.  These  files  can  be  either  manual  or 
autcnated. 

•  External  System.  These  are  represented  in  the  data  flow  diagrams  as  rectangles.  These 
are  organizations,  agencies,  companies,  etc.  with  which  there  is  some  functional 
interaction  during  a  process,  i.e.,  data  is  provided  or  received  from  them. 
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DATA  FLOW  DIAGRAMS 
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Figure  3:  Develop  Training 


Interactive  Courseware 


Figure  3.4. 1 :  Develop  Audio 


Define  Areas 
of  Inflection 


Figure  3.4.2:  Develop  Sound  Effects/Music 
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Figure  3.4.3:  Develop  Graphic/Animation 


Graphic  Files 
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Video 


Figure  3.5.3:  Video  Production 


Figure  3.5.1:  Produce  Audio 
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gure  3.5.2:  Produce  Sound  Effects/Music 
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PROCESS  DESCRIPTIONS 
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NOTE 

The  numbering  of  each  process  description  matches  that  used  in  the  data  flow  diagrams. 
Whenever  the  description  of  a  process  conforms  to  traditional  Instructional  Systems 
Development  (ISD),  AFP  50-58  is  cited  as  the  source  of  information  regarding  the  process. 
Recently  that  approach  has  been  modified  by  AFP  50-68,  but  no  attempt  has  been  made  to 
correlate  each  document  or  reconcile  differences. 


Process  Descriptions 

0.  Instructional  Systems  Development  (ISD).  The  standard  Air  Force  and  military  practice  for 
the  development  of  training.  See  AFP  50-58,  Volume  I:  Introduction,  and  AFM  50-2, 
Instructional  Systems  Development.  In  our  model  ISD  has  five  sub-processes  as  listed  below. 
Since  multimedia  development  is  almost  entirely  concerned  with  step  3  Develop  Training,  the 
model  reflects  it  by  expanding  that  node. 

1 .  Analyze  Training  Requirements 

2.  Design  Objectives  and  Tests 

3.  Develop  Training 

4.  Implement  Training  Program 

5.  Evaluate  Process  and  Products 

1.0  Analyze  Training  Requirements.  Training  requirements  analysis  can  be  a  multi-step  process 
(see  AFP  50-58,  Volume  II).  While  multimedia  development  does  not  normally  affect  the 
performance  of  most  of  these  steps,  there  are  considerations  which  should  be  taken  into  account 
because  of  multimedia  capabilities.  We  have  not  listed  these  considerations  here,  although  most 
can  be  found  in  the  Lessons  Learned  and  Critical  Incidents  sections  of  this  report. 

The  multimedia  development  process  begins  with  analyzing  the  training  problem  or 
conditions.  The  multimedia  developer  brings  to  the  analysis  phase  his  or  her  academic  and 
personal  background  (including  creativity)  and  previous  experience  with  multimedia 
dev  pment  and  training  technologies.  Documentation  including  a  proposal,  statement  of  work, 
and  other  customer-supplied  documents  provide  an  overview  of  the  problem,  status  of  current 
training,  and  proposed  solutions.  Extensive  communication  with  subject  matter  experts  (SMEs) 
is  very  important  during  this  time.  Two  important  documents  produced  during  this  stage  are  the 
statement  of  user  requirements  based  on  the  training  requirements  analysis,  and  specification  of 
the  training  system,  derived  from  analyzing  the  requirements  of  operational  needs.  These  play  a 
major  role  in  guiding  the  production  of  multimedia  elements  during  all  stages  of  ISD.  The 
multimedia  developer  develops  an  overview  describing  the  training  system  by  listing  concepts  to 
be  learned  and  how  they  will  be  portrayed.  This  provides  the  multimedia  development  team  with 
a  tool  to  define  the  scope  and  content  of  the  system  and  serves  as  a  basis  for  discussion  with 
SME(s).  The  system  requirements  which  need  to  be  analyzed  during  this  phase  include  hardware 
and  software  parameters.  A  milestone  chart  is  also  produced  showing  the  different  phases  and 
goals  of  training  development. 
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2.0  Design  Objectives  and  Tests.  The  design  of  objectives  and  tests  is  not  normally  part  of  the 
multimedia  process.  The  details  of  this  process  can  be  found  in  AFP  50-58,  Volume  m.  When 
designing  objectives  and  tests  the  capabilities  of  multimedia  should  be  taken  into  consideration. 
Again,  we  have  not  detailed  these  considerations  here.  They  may  be  found  elsewhere  in  this 
report. 


The  design  phase  of  a  multimedia  project  encompasses  interactive  and  content  design, 
graphic  and  production  design,  software  design,  and  instructional  design.  Using  information 
from  the  SME  obtained  during  analysis  and  specifications  of  the  training  system  requirements, 
the  instructional  developer/multimedia  developer  plan  the  organization  and  appearance  of  the 
lessons.  The  multimedia  developer  uses  his  or  her  knowledge  about  the  technical  capabilities  of 
the  multimedia  environment  to  design  an  effective  student  interface,  and  to  present  and  test 
training  concepts  and  scenarios.  For  example,  the  DVI  training  system  can  accommodate  a  one- 
quarter  screen  window  to  display  full  motion  video  to  most  effectively  illustrate  visual 
references.  These  ideas  and  strategies  are  reflected  in  storyboards  which  are  produced  during 
development.  Multimedia  elements  for  design  include  video,  still  image,  graphic,  audio, 
animation,  music,  and  sound  effects. 

A  multimedia  developer  accomplishes  many  traditional  software  development  tasks. 
However,  a  critical  trait  of  an  effective  multimedia  developer  is  to  view  the  project  from  a 
multimedia  perspective.  A  multimedia  perspective  is  defined  as  being  able  to  apply  the  technical 
capabilities  of  a  multimedia  environment  to  the  application's  requirements.  The  developer 
should  know  general  technical  ramifications  and  the  resources  needed  for  its  accomplishment. 
The  multimedia  developer  needs  to  be  creative,  as  well  as  have  knowledge  of  interactive 
courseware  design,  interface  or  screen  design,  video,  audio,  graphics,  and  color  selection.  He  or 
she  should  have  a  technical  understanding  of  amount  of  time  needed  for  development  and 
resource  allocation.  During  the  design  stage  the  multimedia  developer  has  many  opportunities  to 
apply  multimedia  solutions,  and  to  consider  contingency  solutions  that  support  the  constraints  of 
time  and  funding. 

3.0  Develop  Training.  The  major  activities  of  developing  training  are  described  in  AFP  50-58, 
Volume  IV.  This  model  has  elaborated  the  activities  into  6  major  processes  which  are  normally 
performed  in  the  development  of  interactive  courseware  as  indicated  below.  The  focus  of  the 
model  is  on  3.4  Multimedia  Development  and  3.5  Multimedia  Production. 


3. 1 .  Develop  Instructional  Strategies 

3.2.  Select  Media 

3.3.  Develop  Storyboards 

3.4.  Multimedia  Development 

3.5.  Multimedia  Production 

3.6.  Validate  Courseware 


During  development  a  script/narration  document  is  developed  which  integrates  the 
multimedia  elements  and  provides  the  overview  and  sequence  for  the  lesson.  Finally,  a  lesson 
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prototype  is  constructed  which  provides  a  model  from  which  subsequent  lessons  or  frames  are 
patterned. 

3.1  Develop  Instructional  Strategies.  Using  the  objectives  and  tests  developed  in  the  design 
phase,  the  instructional  designer  selects  instructional  strategies  which  most  effectively  present  the 
objectives  to  the  students.  Instructional  strategies  are  annotated  on  the  rudimentary  curriculum 
taking  form  with  the  objectives  and  tests.  For  a  complete  explanation  see  AFP  50-58,  Volume 
IV. 

3.2  Select  Media.  The  instructional  design  team  assesses  the  instructional  strategies  assigned  to 
each  objective  and  selects  a  presentation  medium  for  the  objective.  Media  models  are  normally 
used  in  selecting  media.  Cost  and  other  considerations  are  taken  into  account  in  defining  the 
media  mix.  It  is  highly  beneficial  for  an  experienced  multimedia  developer  to  provide  input 
during  media  selection  to  capitalize  on  those  visualization  technologies  which  multimedia  offers. 
Many  objectives  can  benefit  from  multimedia  and  they  should  be  identified  here.  For  more 
explanation  see  AFP  50-58,  Volume  IV. 

3.3  Develop  Storyboards.  Once  media  have  been  selected,  storyboards  are  developed  for  those 
objectives  designated  for  interactive  courseware  development.  The  instructional  developer  uses 
existing  job  or  task  data  in  the  form  of  tech  orders,  job  books,  manuals,  etc.,  and  the  input  of 
SMEs,  if  the  developer  is  not  a  technical  expert,  to  develop  a  storyboard  for  the  objective(s). 
Storyboards  form  the  basis  for  further  work  during  multimedia  development  and  production.  For 
more  explanation  see  AFP  50-58,  Volume  IV. 

3.4  Multimedia  Development.  During  multimedia  development  storyboards  are  converted  into 
interactive  courseware.  The  multimedia  development  team  uses  audio  and  video  files,  and  an 
authoring  language  or  system  to  develop  individual  lessons.  Selection  of  multimedia  elements 
for  production  depends  on  the  specific  content  and  function  of  the  lesson.  Several  factors  arc 
considered:  1)  relative  importance  or  prominence  of  the  multimedia  elements  (for  example, 
highest  priority  is  assigned  for  producing  full  motion  video  if  full  motion  video  is  required  to 
teach  critical  concepts);  2)  availability  or  stage  of  development  of  component  materials;  3) 
availability  of  software  tools  and  personnel,  and;  4)  overall  complexity  level  (for  example,  if 
full  motion  video  footage  is  required  that  involves  lengthy  or  hard-to-schedule  shots,  it  is 
scheduled  early  in  production).  The  multimedia  developer  reviews  training  requirements, 
storyboards,  and  the  script/narration  to  prioritize  production  of  multimedia  elements.  The  major 
activities  of  multimedia  development  are: 

3.4.1  Develop  Audio 

3.4.2.  Develop  Sound  Effects/Music 

3.4.3.  Develop  Graphic/Animation 

3.4.4.  Develop  Still  Image 

3.4.5.  Develop  Motion  Video 

3.4.6.  Author  Lesson 

3.4.7.  Define  Standards 
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3.4.1  Develop  Audio.  Developing  audio  for  multimedia  consists  of  using  the  storyboards  and 
standards  to  develop  an  audio  script  which  can  be  used  to  produce  the  required  audio.  The  major 
activities  of  developing  audio  are: 

3.4. 1 . 1 .  Select  Audio  Elements 

3.4. 1 .2.  Develop  Script  for  Narration 

3.4. 1 .3.  Identify  Voice  Types 

3.4. 1 .4.  Define  Areas  of  Inflection 

3.4. 1 .5.  Review  Final  Script 

3.4. 1.1  Select  Audio  Elements.  The  multimedia  developer  reviews  the  storyboard  for  a  lesson  to 
determine  the  specific  audio  elements  required  to  support  the  instructional  strategy.  Audio 
elements  required  are  highlighted.  Audio  elements  can  consist  of  the  mainline  narration  for  the 
lesson,  voice  effects  for  special  purposes  like  help,  narration  of  feedback,  narration  of  optional 
materials,  etc. 

3.4. 1.2  Develop  Script  for  Narration.  As  audio  elements  are  selected  for  use,  they  are  developed 
into  a  narration  script.  In  general,  narration  in  multimedia  applications,  directs  the  student's 
mainline  progress  through  the  lesson.  The  script  for  narration  should  take  into  account  common 
student  errors  and  point  them  out.  The  multimedia  developer  should  consider  the  use  of  text  in 
addition  to  narration.  Bulletized  text  can  often  be  used  to  support  a  narrative  script.  The 
multimedia  developer  should  develop  scripts  for  positive  and  negative  feedback,  remediation  and 
help.  Each  element  of  the  storyboard  may  have  several  narrative  components  associated  with  it. 

3.4. 1.3  Identify  Voice  Types.  The  multimedia  developer  should  decide  whether  to  use  a  male  or 
female  voice,  a  single  voice  or  a  combination  of  voices  aid  what  purpose  each  type  of  voice 
serves.  Each  voice  type  has  definite  characteristics  and  advantages  and  disadvantages.  The 
audio  script  should  reflect  the  selection  of  voices  and  maintain  consistency  throughout  the 
application  in  order  not  to  distract  the  student. 

3.4. 1.4  Define  Areas  of  Inflection,  etc.  The  multimedia  developer  should  specify  on  the 
narrative  script  areas  of  inflection,  emphasis,  and  other  special  voice  effects  which  are  required  to 
accomplish  the  lesson  objectives.  Individual  creativity  can  be  a  major  factor  in  the  proper 
application  of  such  features. 

3.4. 1.5  Review  Final  Script.  When  the  narrative  script  is  complete  it  is  reviewed  for  correctness, 
to  ensure  complete  content  coverage,  to  make  sure  that  proper  language  is  used  (i.e.,  in  keeping 
with  what  the  user  is  expecting),  to  make  sure  proper  pronunciation  is  emphasized,  and  to  ensure 
that  all  narrative  elements  support  the  lesson  objectives. 

3.4.2  Develop  Sound  Effects/Music.  Sound  effects  and  music  contribute  to  the  instructional 
effectiveness  of  a  multimedia  lesson.  Sound  effects  are  obtained  from  commercial  vendors 
already  recorded  for  use  in  other  applications,  or  they  are  developed  on  location  or  in  the  studio. 
The  major  activities  of  developing  sound  effects  and  music  are: 
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3.4.2. 1  Define  Special  Areas  for  Effects  Support 

3.4.2.2.  Develop  Script  for  Sound  Effects/Music 

3.4.2.3.  Check  Copyright 


3.4.2. 1  Define  Special  Areas  for  Effects  Support.  The  multimedia  developer  will  check  the 
storyboards  to  determine  what  sound  effects  or  music  are  required.  Some  sound  effects  such  as 
machine  noise,  alarm  sounds,  danger  signals,  etc.  may  be  absolutely  necessary  in  support  of  the 
lesson  objectives.  Some  other  sound  effects  or  music  may  be  aesthetically  pleasing  to  the 
student,  or  they  may  enhance  the  lesson  objective  by  promoting  student  attention.  Each  sound 
element  should  be  selected  based  on  a  legitimate  instructional  reason. 

3. 4.2.2  Develop  Script  for  Sound  Effects/Music.  As  sound  effects  or  music  requirements  are 
identified  from  the  storyboards,  a  sound  effects  script  is  developed  to  catalog  the  various  effects 
needed. 

3. 4.2.3  Check  Copyright.  When  using  commercially  available  sound  effects  or  music,  legal 
restrictions  md  copyright  must  be  checked.  If  copyrighted  material  must  be  used,  a  copyright 
release  or  permission  must  be  obtained.  Caution  must  be  exercised  since  royalties  may  need  to 
be  paid  to  some  copyright  holders. 

3.4.3  Develop  Graphic/Animation.  The  computer  industry  has  been  involved  with  graphics  and 
its  production  processes  before  the  advent  of  multimedia.  New  data  are  available  and  experience 
is  being  documented  abcut  3-D  graphics  and  animation  production.  Technical  issues  associated 
with  the  integration  of  digital  images,  video  and  graphics,  3-D  graphics  and  animation  are 
challenges  for  the  multimedia  developer.  The  integration  of  these  capabilities  is  facilitated  by  a 
multimedia  developer  who  has  technical  experience  with  the  requirements  needed  for  multimedia 
elements  and  the  performance  improvements  of  the  new  technology  and  PCs. 

The  multimedia  developer  always  has  new  and  advanced  technologies  and  capabilities  to 
choose  from.  The  use  of  various  multimedia  capabilities  provides  opportunities  to  develop 
effective  training  applications.  Animation  provides  a  useful  and  effective  visual  aid,  and  newly 
affordable  3-D  animation  introduces  further  capabilities  that  could  satisfy  training  requirements. 

There  should  be  a  special  effort  dedicated  to  the  production  of  the  student  interface. 
Production  of  the  graphical  user  interface  is  accomplished  by  the  use  of  a  graphics  or  authoring 
software  package.  Most  of  the  interface  elements  are  produced  in  graphics  or  still  image 
production  sections.  Production  of  the  interface  should  evolve  from  brainstorming  sessions  with 
the  user  to  produce  an  interface  prototype,  which  can  finally  be  integrated  into  a  multimedia 
application.  The  multimedia  development  team  may  need  to  adjust  or  edit  the  interface  in  mid¬ 
development;  it  can  be  considered  a  living  entity  which  is  a  direct  extension  of  the  user's 
requirements.  The  interface  and  the  support  for  system  navigation  must  support  the  user 
completely.  The  user  should  not  have  to  spend  much  time  figuring  out  how  to  use  the  system 
since  an  intuitive  theme  is  usually  more  effective  for  training. 
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The  major  activities  involved  in  graphic  development  are: 

3.4.3. 1  Develop  Graphic  List 

3.4.3.2.  Develop  Graphic 

3.4.3.3.  Animate  Graphic 

3.4.3.4.  Edit  Video  Image 

3.4.3.5.  Check  Graphic 

3.4.3. 1  Develop  Graphic  List.  The  multimedia  development  team  must  be  aware  of  the 
capabilities  of  the  graphics  package  which  is  being  used.  Storyboards  are  reviewed  to  determine 
the  number  and  type  of  graphics  which  need  to  be  developed.  If  video  images  can  be  used 
instead  of  creating  new  graphic  images,  they  should  be  identified.  Still  images  which  have  not 
already  been  shot  need  to  be  added  to  the  shot  sheet.  The  multimedia  developer  must  ensure  that 
the  video  images  to  be  used  are  in  the  proper  format  to  be  modified  (cropped,  cleaned  up,  etc.)  by 
the  graphics  package.  Graphics  in  the  storyboards  which  will  not  utilize  video  images  should  be 
identified  for  creation.  If  graphics  are  to  be  used  more  than  once  in  the  application,  they  should 
be  cataloged  and  each  appearance  noted.  Whenever  animation  is  required  by  the  storyboards,  the 
requirement  should  be  annotated  on  the  graphic  list.  Considerations  which  should  be  taken  into 
account  in  determining  to  use  a  graphic  or  a  video  image,  an  animation  or  a  motion  video  are:  1) 
instructional  soundness  of  the  approach,  2)  ease  of  creation,  3)  instructional  effectiveness,  and;  4) 
cost  in  resources. 

3.4.3.2  Develop  Graphic.  The  graphic  artist  should  work  from  the  graphic  list  and  storyboards 
to  create  the  graphic.  The  graphic  software  must  be  capable  of  the  various  formats  required  by 
the  application.  Graphics  and  images  should  be  in  a  single  color  palette.  Once  the  graphic  is 
complete  the  file  should  be  named  according  to  conventions  agreed  upon  by  the  multimedia 
development  team.  Finally,  log  the  graphic. 

3.4.3.3  Animate  Graphic.  The  multimedia  developer  always  has  new  and  advanced  technologies 
and  capabilities  to  choose  from.  The  use  of  various  multimedia  capabilities  provides 
opportunities  to  develop  effective  training  applications.  Animation  provides  a  useful  and 
effective  visual  aid,  and  newly  affordable  3-D  animation  introduces  further  capabilities  that  could 
satisfy  training  requirements.  Animation  production  is  very  similar  to  that  of  graphics 
production.  The  graphic  artist  works  from  the  storyboards  and  the  graphics  list  to  determine  the 
type  and  level  of  fidelity  (3-D,  stick  figure,  etc.)  required  in  the  animation,  and  makes  sure  the 
animation  software  supports  the  application.  The  animation  file  should  be  named  in  accordance 
with  the  conventions  agreed  upon  for  the  project.  When  the  animation  has  been  completed,  log 
the  animation  in. 

3.4.3.4  Edit  Video  Image.  When  video  images  are  used  it  is  frequently  necessary  to  edit  the 
image  to  meet  the  storyboard  requirements.  Raw  video  images  (either  still  frame  or  motion 
video)  can  be  used.  The  graphic  artist  and  multimedia  developer  determine  how  the  image 
selected  should  be  edited  to  fit  the  application.  The  image  may  need  to  be  cropped,  re-sized, 
touched  up,  have  adjustments  made  in  the  color  palette,  or  other  editing.  When  the  graphic  is 
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complete,  the  file  is  named  in  accordance  with  the  project  file  naming  conventions.  Log  the 
graphic. 

3.4.3.5  Check  Graphic.  A  quality  control  check  should  be  made  of  each  graphic  created  prior  to 
inserting  it  into  the  application.  When  the  programmer  tests  the  graphic  in  the  application  it 
should  be  as  ready  as  possible  so  that  the  multimedia  lesson  appears  as  near  to  the  finished 
product  as  possible.  If  minor  adjustments  are  required  in  the  graphic,  they  should  be  noted  and 
the  graphic  sent  back  to  the  graphic  artist  for  correction.  Log  all  actions  in  the  graphic  log. 

3.4.4  Develop  Still  Image.  The  need  for  graphics  can  easily  be  supported  or  replaced  by  digital 
video's  still  image  capability.  Photo  quality  imagery  is  effective  and  can  support  an  application's 
need  for  realism.  Additionally,  labor  associated  with  graphic  development  is  time  consuming 
and  costly.  However,  technical  issues  such  as  the  size  of  the  image,  color  palette  compatibility, 
or  editing  time  may  require  graphics  instead.  The  capture  (also  called  digitization),  and  editing 
(production)  of  still  images  is  facilitated  if  the  multimedia  developer  is  aware  of  still  image 
requirements  during  video  shoots.  Many  images  can  be  obtained  cost  effectively  at  these  times. 
The  cost  effective  planning  and  use  of  video  production  schedules,  including  setting  up 
equipment,  transportation,  and  local  coordination  of  activities  is  essential. 

3.4.5  Develop  Motion  Video.  A  shot  sheet  is  a  checklist  of  conditions  for  shooting  new  video 
footage.  A  shot  sheet  is  developed  by  the  multimedia  developer  for  directing  the  video 
technician.  Several  resources,  developed  by  the  multimedia  developer  working  with  the  SME 
during  the  analysis  and  design  phases,  are  used  to  produce  the  list  of  required  scenes  for  the 
lesson,  including  specific  directions  for  shooting  each  scene.  These  resources  include 
storyboards,  user  training  requirements,  and  existing  video.  Existing  video  (often  provided  by 
the  customer)  is  also  reviewed  for  content  and  quality  in  order  to  plan  which  new  video 
sequences  are  required.  The  shot  sheet  is  designed  with  the  video  production  schedule  in  mind 
since  it  specifies  the  cost,  time,  personnel,  equipment,  and  other  resources  required  to  produce 
full  motion  video. 

Design  of  the  shot  sheet  is  constantly  checked  in  order  to  make  sure  that  the  directions  for 
all  the  sequences  are  clear  and  complete.  Alternative  directions,  such  as  trying  out  a  different 
lighting  arrangement  in  a  scene,  are  also  included.  The  multimedia  developer  also  specifies  in 
the  shot  sheet  which  still  images  and  graphics  are  obtained  from  full  motion  video.  This  revised 
and  edited  shot  sheet  is  ready  to  use  by  the  video  technician. 

3.4.6  Author  Lesson.  Authoring/programming  is  the  process  of  integrating  the  multimedia 
elements  into  the  lesson,  and  includes  all  the  technical  specifications  and  functional  descriptions 
from  analysis,  design,  and  production.  This  involves  scripting,  or  writing  programs  to 
accomplish  specific  functions  in  the  lesson.  Working  from  the  storyboards,  script/narration,  and 
shot  sheets  (or  multimedia  element  lists),  digitized/edited  multimedia  elements  files  are 
programmed  into  the  training  system.  Problem  solving  techniques  for  software  and  hardware 
integrate  the  different  components;  knowledge  about  software  and  hardware  capabilities  and 
compatibility's  is  extremely  important.  Some  editing  or  tweaking,  a  series  of  subtle  adjustments 
to  improve  the  system,  is  also  performed  during  this  stage.  An  example  of  tweaking  is  modifying 


82 


the  justification  of  tc:u  in  order  to  improve  the  appearance  of  a  screen.  The 
programming/authoring  stage  results  in  a  complete  programmed  lesson  integrating  all  of  the 
multimedia  elements,  branching  options  and  levels  of  interactivity,  and  instructional  design.  The 
lesson  is  now  ready  for  customer  review  and  testing. 

3.4.7  Define  Standards.  At  the  beginning  of  the  multimedia  project  the  multimedia  team  should 
develop  a  set  of  standards  that  can  be  achieved  within  the  scope  of  the  software  packages  being 
used  and  in  keeping  with  the  requirements  of  the  lesson  objectives.  Standards  should  be 
developed  to  determine  file  naming  conventions,  video  formats  used,  color  palette,  scripting  and 
storyboarding  format,  screen  layout,  text  fonts,  and  all  other  aspects  which  affect  the  quality  and 
consistency  of  the  finished  product.  A  standards  document  should  be  developed  and  provided  to 
each  member  of  the  multimedia  development  team  for  use  during  the  project.  The  standards 
document  may  be  called  a  format  guide,  standard,  or  any  other  name  which  designates  its 
purpose  but  it  is  a  working  document  which  must  be  accessible  to  all  of  the  multimedia  team 
members.  Whenever  standards  do  not  serve  the  best  interests  of  the  training  system,  they  can  be 
changed,  but  ensure  that  later  versions  of  a  standard  are  retroactively  applied  to  all  of  the 
courseware  which  has  been  developed  to  date. 

3.5  Multimedia  Production.  The  critical  difference  between  traditional  courseware  development 
and  that  of  multimedia  development  is  production.  Multimedia  is  composed  of  elements  used  by 
the  movie,  video,  radio  and  television  industries.  These  elements  need  to  be  designed  and 
produced  for  use  in  the  multimedia  application.  A  multimedia  developer  can  rely  on  industry 
professionals  for  the  well-established  and  effective  design  and  production  techniques  for  these 
elements.  The  multimedia  developer  must  understand  the  ramifications  of  using  these  elements 
on  the  computer  in  an  interactive  environment.  What  is  effective  on  the  large  motion  picture 
screen  or  even  on  the  moderate  size  screens  used  for  videotape  viewing  may  not  work  well  when 
presented  on  a  computer  monitor  or  within  a  window. 

The  delivery  platform  has  inherent  qualities  which  affect  development  of  a  training 
system.  Good  qualities  of  the  delivery  platform  must  be  exploited  in  a  manner  that  supports 
effective  training  while  bad  qualities  must  be  avoided  or  worked  around.  Creative  and  technical 
flexibility  are  important  qualities.  It  is  at  this  point  during  development  that  the  multimedia 
developer  is  able  to  make  decisions  on  the  fly  that  affect  multimedia  elements,  training 
effectiveness,  technical  issues,  and  cost.  Even  though  the  multimedia  developer  may  rely  on 
industry  professionals  for  their  technical  expertise,  the  responsibility  for  producing  an  effective 
training  application  is  still  the  multimedia  developer's.  A  basic  knowledge  of  the  user,  subject, 
delivery  environment,  and  learning  theory  must  be  applied  during  the  design  and  production  of 
each  multimedia  element. 

The  major  activities  of  multimedia  production  are: 

3.5.1  Produce  Audio 

3.5.2  Produce  Sound  Effects/Music 

3.5.3  Video  Production 

3.5.4  Digitize  Video  and  Audio 
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3.5.5 


Edit  Video  and  Audio 


Identification  of  the  multimedia  elements  for  production  originates  during  the 
multimedia  development  phase.  Storyboards  and  scripts  show  where  and  how  the  multimedia 
elements  are  organized  and  integrated  in  the  lesson.  An  overall  production  schedule  is  devised 
which  describes  in  detail  specific  sequencing  and  step-by-step  instructions  for  producing  the 
individual  multimedia  elements:  video,  still  images,  graphics,  animation,  audio,  music,  and 
sound  effects.  Producing  each  multimedia  element  involves  using  the  information  from  analysis 
and  design,  such  as  user  training  requirements,  to  ensure  that  the  elements  are  used  consistently 
and  effectively  within  the  lesson,  for  example,  standardization  of  text  location  or  an  option  box 
which  gives  students  a  choice  about  how  to  proceed.  Storyboards  and  the  script/narration  are 
also  used  as  the  major  documents  to  guide  multimedia  production,  since  they  describe  and 
display  the  content  and  organization  of  the  lesson.  Production  includes  digitizing  (also  called 
capturing)  and  editing  the  multimedia  files  so  that  they  are  ready  for  authoring  or  programming. 

3.5.1  Produce  Audio.  The  multimedia  developer  integrates  audio,  music,  and  special  effects  in 
an  application  to  support  audio  and  sound  requirements.  The  production  of  the  sound  component 
of  the  multimedia  application  has  specific  technical  issues  associated  with  it.  For  example,  if 
voice  talent  is  used  in  the  application,  it  should  be  realistic.  The  narrator  should  use  a  vernacular 
familiar  to  the  user.  Voices  should  be  recorded  at  a  constant  volume  level,  and  back  ground 
noise  should  be  completely  eliminated.  If  both  male  and  female  voice  talent  are  used,  make  sure 
both  voices  are  being  recorded  at  the  same  volume  levels  and  that  such  serves  a  specific  purpose. 
The  production  of  the  sound  elements  can  go  first  to  tape  then  to  digital  storage  device,  or 
directly  to  the  digital  storage  device.  In  either  case,  it  is  important  to  log  the  data.  The  major 
activities  of  producing  audio  are: 


3.5.1. 1 

Identify  Recording  Needs 

3.5. 1.2 

Obtain  Audio  Resources 

3.5.1.3 

Prepare  Recording  Equipment 

3.5.1.4~ 

Coach  Talent 

3.5. 1.5 

Record  Audio 

3.5.1.6 

Check  Recording 

3.5. 1.7 

Log  Audio  Data 

3.5. 1.1  Identify  Recording  Needs.  The  multimedia  production  team  works  from  the  audio  script 
to  identify  the  various  components  necessary  for  recording  the  audio  narration,  e.g.,  the  number 
and  kinds  of  voices  to  be  used,  what  part  each  voice  reads,  inflection  and  voice  modulation 
requirements,  and  other  factors  concerned  with  recording  the  narration.  If  additional  resources 
are  required  obtain  them  prior  to  recording. 

3.5. 1.2  Obtain  Audio  Resources.  If  talent,  microphones,  special  tapes,  or  other  products  are 
needed  they  should  be  obtained  prior  to  beginning  the  recording  session.  Order(s)  for  the 
product(s)  should  be  sent  in  time  to  ensure  that  the  production  schedule  can  be  met. 
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3.5. 1.3  Prepare  Recording  Equipment.  Recording  equipment  should  be  checked  prior  to 
beginning  the  recording  session.  It  is  always  advisable  to  test  the  recording  equipment, 
microphones,  studio  acoustics,  and  other  equipment  with  the  talent  who  will  be  reading  the 
narration.  Special  requirements  should  be  tested  to  ensure  they  are  ready  for  operation. 

3.5. 1.4  Coach  Talent.  All  talent,  whether  experienced  or  not,  will  require  some  coaching  prior 
to  recording  the  narrative  segments.  Provide  each  voice  talent  with  the  portion  of  the  script  that 
you  want  them  to  read,  point  out  places  of  emphasis,  voice  modulation  patterns  needed, 
inflection  requirements,  and  how  their  portions  may  integrate  with  other  voices  and  other 
multimedia  elements.  When  the  talent  is  prepared,  try  out  the  script  once  to  make  sure  that  they 
can  read  what  has  been  written,  and  that  what  has  been  written  in  the  audio  script  makes  sense 
when  read  aloud. 

3.5. 1.5  Record  Audio.  The  narration  should  be  based  on  the  audio  script.  If  sections  of  the 
script  do  not  sound  like  the  script  or  storyboards  had  intended,  make  changes  and  record  the 
changes  on  the  script.  While  narration  is  most  frequently  recorded  to  tape  first,  then  digitized, 
this  is  not  always  necessary.  If  the  recording  is  not  being  done  in  a  professional  studio,  care  must 
be  taken  to  ensure  that  extraneous  noises  do  not  interfere  with  the  recording  session.  The 
narration  talent  should  be  located  so  that  they  do  not  have  to  strain  or  move  in  and  out  from  the 
microphones.  The  recording  equipment  should  be  placed  such  that  it  provides  a  constant  level  of 
audio  input  from  the  speakers),  unless  stronger  voice  modulation  is  necessary  (as  indicated  on 
the  script). 

3.5. 1.6  Check  Recording.  Once  the  recording  session  has  been  completed,  check  the  recording 
to  make  sure  that  it  says  what  the  script  calls  for,  that  it  sounds  natural  and  clear,  and  that  no 
strange  accents  or  pronunciation  interferes  with  understanding.  If  there  are  problems  with  the 
recording,  point  them  out  to  the  talent  and  re-record  on  the  spot  until  the  problems  are  corrected. 

3.5. 1.7  Log  Audio  Data.  After  the  recording  session  has  been  completed,  record  the  pertinent 
information  about  each  segment  in  the  audio  log. 

3.5.2  Produce  Sound  Effects/Music.  The  major  activities  involved  in  producing  sound  effects 
and  music  are: 

3.5.2. 1  Identify  Recording  Needs 

3.5.2.2.  Obtain  Sound  Effects  or  Music  Source 

3.5.2.3.  Prepare  Recording  Equipment 

3.5.2.4.  Record  Sound  Effects/Music 

3.5.2.5.  Check  Recording 

3. 5.2.6.  Log  Sound  Effects  Data 

3.5.2. 1  Identify  Recording  Needs.  The  multimedia  production  team  works  from  the  shot  sheet  to 
identify  the  various  components  necessary  for  recording  the  music  or  sound  effects.  If  the  sound 
effects  are  to  be  produced  by  the  team  either  at  the  work  location  or  in  the  studio,  the  team  must 
make  sure  that  the  equipment,  tools  and  conditions  necessary  to  produce  the  sounds  will  be 
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available  during  recording.  If  the  sound  effects  or  music  to  be  recorded  is  not  to  be  taken  from 
live  components,  it  will  be  necessary  to  obtain  them  from  some  source,  e.g.,  record,  tape,  disc, 
etc.  Any  special  recording  equipment  requirements  should  be  identified  and  taken  care  of  prior 
to  setting  up  the  recording  session. 

3.5.2.2  Obtain  Sound  Effects  or  Music  Source.  If  music  or  sound  effects  are  used  from  an 
existing  source,  the  team  must  identify  what  music  or  sound  effects  are  needed,  which 
commercial  source  can  provide  that  sound  effect  or  music,  and  order  the  recording  from  a 
commercial  vendor  if  a  copy  is  not  already  available  to  the  team. 

3.5.2.3  Prepare  Recording  Equipment.  Check  to  see  that  the  right  type  of  recording  equipment 
is  available  to  produce  and  record  the  sound  effects  or  music.  Check  out  the  equipment  at  this 
time,  i.e.,  prior  to  the  first  attempt  at  actual  recording  for  use.  If  there  is  a  problem  with  the 
equipment  it  should  be  taken  care  of  prior  to  starting  the  recording  session.  When  the  recording 
equipment  and  sound  effects  production  equipment  is  ready  it  is  time  to  record. 

3.5.2.4  Record  Sound  Effects/Music.  The  actual  recording  the  sound  effects  or  music  should  be 
based  on  the  sounds  called  for  in  the  sound  effects  script.  As  various  sound  effects  segments  are 
recorded  the  team  should  record  pertinent  information  about  each  one,  e.g.,  duration,  brief 
description  about  what  is  recorded,  start/stop  locations,  file  or  tape  recorded  on,  etc. 

3.5.2.5  Check  Recording.  A  quality  check  should  be  performed  on  each  segment  immediately 
after  it  has  been  recorded.  If  there  are  problems,  or  if  the  recording  does  not  meet  quality 
standards,  re-record  it. 

3.5.2.6  Log  Sound  Effects  Data.  As  sound  effects  and  music  segments  are  recorded  and 
checked,  they  should  be  entered  into  the  audio  log.  Record  pertinent  information  about  each  one, 
e.g.,  duration,  brief  description  about  what  is  recorded,  start/stop  locations,  file  or  tape  recorded 
on,  special  notes  about  the  recording,  storyboard  to  which  it  relates,  etc. 

3.5.3  Video  Production.  The  multimedia  developer  must  realize  that  the  use  of  multimedia  does 
not  guarantee  effective  training,  although  creative  use  of  video  can  be  an  essential  element  of  a 
good  training  system.  The  multimedia  developer  must  be  concerned  with  the  way  video  is  shot, 
the  size  of  the  image  that  will  appear  on  the  computer  screen,  and  potentially  distracting 
movements  of  camera  and  subject.  Technical  considerations,  such  as  size  of  digital  video  file(s), 
maintenance  of  digital  files  during  development,  and  access  to  digital  files  in  the  application,  are 
also  other  considerations.  As  digital  video  products  increase  in  capabilities,  editing  software, 
compression,  and  fidelity  and  their  effects  on  the  application,  are  the  main  issues.  For  example, 
the  availability  of  editing  software  allows  for  quicker  and  more  cost  effective  updates  to  video 
used  in  an  application  allowing  for  more  responsive  development  and  a  greater  opportunity  to 
satisfy  user  requirements  (even  as  they  change).  Compression  is  usually  associated  with  the  size 
of  the  digital  video  file.  As  compression  improves  so  does  the  amount  of  data  that  can  be  stored 
on  a  computer  for  an  application.  Currently,  developers  need  to  understand  delivery  system 
storage  requirements  and  system  performance  as  they  design  and  produce  for  a  multimedia 
application.  Some  applications  are  not  candidates  for  digital  video  (or  multimedia  using  digital 
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video)  because  the  fidelity  or  quality  requirements  are  not  supported.  As  the  fidelity  of  digital 
video  improves  so  will  the  number  of  applications  that  it  can  effectively  support. 

The  major  activities  of  video  production  are: 


3.5.3. 1 

Establish  Production  Schedule 

3.5.3.2 

Determine  Shoot  Location 

3.5.3.3 

Determine  Transportation  Requirements 

3.5.3.4 

Prepare  Equipment 

3.5.3.5 

Prepare  Set 

3.5.3.6 

Check  Lighting 

3.5.3.7 

Determine  If  Tape  or  Capture 

3.5.3.8 

Shoot  Video 

3.5.3.9 

Check  Video 

3.5.3.10 

Record  Notes  in  Video  Log 

3.5.3. 1  Establish  Production  Schedule.  Using  documents  produced  during  previous  phases, 
such  as  the  milestone  chart,  user  and  system  requirements,  storyboards,  and  narrative  script,  the 
multimedia  developer  establishes  an  overall  production  schedule  which  specifies  a  plan  for  each 
multimedia  element.  Planning  includes  estimating  the  cost,  amount  of  time,  materials  and 
personnel  required  to  complete  each  production  process. 

3.5.3.2  Determine  Shoot  Location.  Several  variables,  such  as  subject  matter,  lighting,  facilities, 
and  equipment  are  considered  when  selecting  a  studio  or  location  shoot.  It  is  usually  preferable 
to  use  a  studio  since  the  environment  can  be  controlled;  however,  it  is  not  always  cost-effective, 
practical,  or  realistic  (e.g.,  planes  flying).  In  some  cases,  legally  binding  permission  must  be 
obtained  before  shooting,  if  shots  involve  shooting  at  a  private  facility.  For  both  studio  and 
location  shoots,  lighting  requirements  are  evaluated  to  enhance  visibility,  visual  depth  and 
attractive  shots. 

3.5.3.3  Determine  Transportation  Requirements.  If  video  shooting  is  to  be  done  on  location  or 
even  at  a  studio  and  transportation  must  be  provided  for  equipment  and  other  props,  these 
arrangements  should  be  made  far  enough  in  advance  so  as  not  to  hold  up  any  of  the  other 
production  activities.  Transportation  to  a  location  may  seem  like  an  insignificant  part  of  the 
overall  video  production  effort,  but  without  it  all  other  production  work  may  not  be  possible. 

3.5.3.4  Prepare  Equipment.  The  multimedia  development  team  needs  to  work  as  cost 
effectively  as  possible.  The  faster  the  team  works  to  get  everything  into  place  the  cheaper  the 
shoot  will  be.  When  a  professional  video  production  crew  is  used,  the  clock  is  always  running. 
In  fact,  the  saying  "time  is  money"  applies  to  all  parts  of  the  multimedia  development  process,  no 
matter  which  personnel  are  involved.  Camera,  tapes,  props,  monitors,  and  other  video 
production  equipment  should  be  itemized  and  ready  prior  to  arriving  at  the  location  or  studio  so 
that  production  will  not  be  held  up  waiting  for  all  the  elements  to  be  assembled. 
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3.5.3.5  Prepare  Set.  When  shooting  on  location  all  of  the  tools  and  equipment  for  producing  the 
video,  and  those  which  are  to  be  included  in  the  video  should  already  be  on  the  set.  Check  to 
make  sure  that  these  are  ready  and  in  working  condition  prior  to  the  recording  session.  For 
example,  things  can  be  overlooked  such  as  equipment  which  needs  to  be  warmed-up  for  several 
hours  prior  to  use.  These  seemingly  insignificant  considerations  can  lead  to  costly  shooting 
delays.  Even  when  shooting  on  location  it  is  necessary  to  prepare  the  set.  Equipment  may  not  be 
in  the  best  location  for  shooting,  i.e.,  lighting  may  be  poor,  or  there  may  not  be  enough  room  for 
camera  and  lights  to  be  positioned  properly.  Check  shooting  locations  prior  to  conducting  the 
shoot;  if  necessary  arrange  for  changes  to  be  made  prior  to  the  scheduled  shooting  time,  then 
check  the  location  again.  When  shooting  in  a  studio  or  other  controlled  environment  all 
equipment  necessary  for  the  shoot  should  be  assembled  and  checked  out  to  ensure  it  is  ready  to 
shoot.  Locations  may  need  to  be  marked  off  for  equipment,  props  and  talent  with  chalk  prior  to 
the  scene  being  recorded.  If  the  set  can  not  be  ready  in  time,  it  may  be  advisable  to  postpone  the 
shoot  to  some  other  time  rather  than  waste  the  time  of  all  the  principals  involved. 

3.5.3.6  Check  Lighting.  Lighting  must  be  checked  to  ensure  that  all  elements  mentioned  in  the 
storyboard  are  visible  to  the  proper  level  of  fidelity.  Lighting  in  the  studio  can  be  controlled  by 
using  reflectors  and  various  lighting  techniques  to  ensure  that  the  right  effect  is  achieved.  When 
shooting  on  location  lighting  conditions  of  natural  light  must  be  understood  and  taken  into 
consideration  in  planning  the  scene.  Overcast  skies  provide  different  lighting  conditions  than 
direct  sunlight.  Talent  may  have  to  be  placed  in  different  locations  because  of  sunlight  which  is 
directly  in  their  eyes.  Lighting  must  be  checked  to  ensure  that  all  objects  in  the  scene  are  clearly 
visible  and  without  glare.  If  glare  is  present,  techniques  must  be  employed  to  reduce  or  eliminate 
the  effects  of  glare.  Shadows  also  pose  a  problem.  Care  should  be  taken  to  ensure  that  shadows 
do  not  detract  from  the  desired  effect  for  a  scene. 

3.5.3.7  Determine  If  Tape  or  Capture.  There  are  special  conditions  a  multimedia  developer 
should  be  aware  of  when  digitizing  and  storing  images  on  the  hard  drive.  Using  the  storage 
capabilities  of  digital  media  creates  a  flexible  environment  for  image  capture.  Various 
condi tionsxan  be  demonstrated  and  experiments  may  be  performed,  the  results  of  which  can  be 
viewed  immediately  in  the  application.  This  is  an  opportunity  that  should  be  exploited  by  the 
developer  in  order  to  allow  the  user  to  see  and  make  decisions  in  a  tangible  way.  Capturing 
images  directly  onto  the  digital  storage  device  or  large  hard  drive  must  be  done  in  a  controlled 
environment.  An  image  log  which  contains  descriptive  notes  and  characteristics  of  the  image  is 
helpful.  Other  special  conditions  include  capturing  in  the  same  color  palette,  at  the  same  aspect 
ratio,  and  with  lighting  and  support  colors  that  are  consistent  with  the  interface. 

If  the  multimedia  developer  decides  to  use  tape  before  capturing  into  the  digital 
environment,  he  or  she  must  consider  these  conditions:  logging,  location,  notes,  lighting,  and 
planning  for  the  digitizing  or  capturing  process.  During  this  process,  the  issues  to  be  concerned 
about  are  similar  to  those  issues  of  capturing  directly  to  digital  storage  device. 

3.5.3.8  Shoot  Video.  The  video  shoot  is  based  on  three  documents—  the  video  production 
schedule,  the  edited  shot  sheets,  and  the  storyboards.  First,  a  production  schedule  is  constructed 
which  considers  logistics  such  as  location,  transportation,  and  permission,  if  required. 
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Equipment  is  tested  to  make  sure  everything  works;  supplies  are  checked.  If  required,  personnel 
are  coached  about  their  roles.  Conducting  a  practice  run-through  video  shoot  helps  to  reduce 
shoot  time,  since  problems  can  be  identified  and  corrected.  After  shooting  new  footage,  the 
multimedia  developer  confirms  on  an  updated  shot  sheet  that  the  scenes  are  complete  (or  if 
scenes  need  to  be  re-shot).  The  updated  shot  sheet  also  includes  extensive  notes  on  the  scene-by¬ 
scene  results  of  the  video  shoot. 

3.5.3.9  Check  Video.  While  recording  the  video,  prior  to  moving  on  to  the  next  scene,  the 
multimedia  developer  should  check  the  video  to  ensure  that  it  meets  standards  and  accomplishes 
the  results  indicated  in  the  storyboards.  The  best  way  of  checking  this  raw  video  is  to  view  it  on 
a  screen.  The  developer  views  the  video  for  content  accuracy,  conformance  with  standards,  and 
general  appearance  if  discrepancies  are  noted  the  multimedia  developer  should  point  them  out  to 
those  concerned  so  that  they  can  make  corrections  and  re-shoot  the  video.  After  the  video  has 
been  recorded  a  quality  check  should  be  made  to  ensure  that  everything  indicated  in  the  shot 
sheet  and  storyboards  has  been  accomplished. 

3.5.3.10  Record  Notes  in  Video  Log.  During  video  production  notes  should  be  kept  about  video 
data  such  as  length  of  segments,  location  on  the  tape,  number  of  tapes  recorded,  and  other  such 
technical  factors.  After  all  video  production  has  been  completed  the  notes  should  be  assembled 
and  any  annotations  made  as  necessary. 

3.5.4  Digitize  Video  and  Audio.  Digitization  is  the  process  of  converting  an  analog  signal  into  a 
digital  signal,  also  known  as  capturing.  All  multimedia  elements  except  for  those  produced 
within  the  computer,  are  digitized  before  being  integrated  into  the  training  system.  The 
multimedia  developer  works  from  the  multimedia  elements  list  to  produce  a  digitization  schedule 
and  a  digitization  sheet  with  updated  file  name,  number,  size,  and  location  information.  The 
digitization  sheet  and  schedule  are  used  to  organize  and  guide  digitization  activities  and  to 
monitor  the  status  of  the  multimedia  elements. 

It  is  preferable  to  digitize  before  editing.  A  digitization  schedule  lists  the  steps  which 
must  be  accomplished  along  with  how  long  each  of  them  will  take.  A  digitization  sheet  is 
produced  from  the  revised  shot  sheet  containing  information  about  the  description  and  location 
of  the  tape  segment,  its  quality,  exact  length,  and  its  destination,  for  both  existing  and  new  full 
motion  video  footage.  Software  capture  modification  can  also  be  used  to  digitally  change  values 
of  the  images. 

3.5.5  Edit  Video  and  Audio.  Editing  is  the  process  of  altering  or  otherwise  improving  the 
appearance  or  organization  of  the  multimedia  elements.  An  example  is  standardizing  the  fades 
and  transitions  between  full  motion  video  clips  in  a  way  that  enhances  the  appearance  of  lesson 
frames  and  also  does  not  distract  students.  The  multimedia  developer  needs  to  edit  each  of  the 
multimedia  elements  in  the  training  application  in  the  appropriate  location,  for  example,  by 
matching  narration  with  full  motion  video  clips  and  text  in  a  particular  lesson  frame. 
Maintaining  extensive  and  precise  documentation  in  the  form  of  storyboards,  and  the  narrative 
script  throughout  this  process,  facilitates  editing  by  illustrating  the  placement,  sequence,  and 
rationale  for  editing  each  element.  The  digitization  sheet  is  used  for  tracking  the  multimedia 
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elements.  As  each  file  is  edited,  a  new  edit  list  is  developed  which  describes  the  modification 
and  new  location  of  the  file  for  authoring/programming  purposes. 

It  is  preferable  to  edit  digitized  video,  although  some  editing  may  occur  prior  to  or  during 
digitization.  The  most  critical  step  is  to  construct  the  edit  video  list  (which  is  the  same  as  the  edit 
decision  list  for  analog  edits)  which  specifies  edit  activities.  The  video  is  reviewed  and  if  the 
quality  is  not  acceptable,  the  multimedia  developer  may  want  to  re-digitize.  In  some  cases,  fades 
and  transitions  may  be  added  during  digitizing.  An  updated  video  log  is  produced.  Edited  video 
segments  or  clips  are  ready  for  programming/authoring  according  to  an  updated  video  log. 

The  editing  process  is  a  significant  part  of  developing  a  multimedia  application.  The 
multimedia  developer  however,  can  often  rely  on  professionals  in  the  industry  to  accomplish  the 
editing  requirements  for  some  multimedia  elements.  However,  the  multimedia  developer  first 
needs  to  edit  each  of  the  multimedia  elements  into  the  application  in  the  appropriate  location. 
This  reinforces  the  need  for  comprehensive  documentation  during  the  development  and 
production  stages.  The  multimedia  developer  needs  to  determine  the  effectiveness  of  each 
multimedia  element.  If  the  training  requirement  has  not  been  met,  the  developer  must  ensure  that 
re-design  and  further  production  is  accomplished  to  meet  those  needs. 

3.6  Validate  Courseware.  Interactive  courseware  is  evaluated  for  consistency,  correctness, 
ability  to  teach  the  objectives,  and  other  factors  during  formative  evaluation.  If  problems  are 
detected  in  the  courseware,  they  are  noted  and  revisions  are  made  to  the  courseware  as  indicated 
by  the  evaluation  plan.  For  more  explanation  see  AFP  50-58,  Volume  IV. 

Testing  and  validation  is  performed  by  users  who  are  experts  in  the  various  domains 
represented  in  training.  This  includes  a  review  of  content,  system  navigation,  and  overall 
appearance  and  feel  of  the  lesson  in  accordance  with  the  storyboards  and  script/narration.  The 
customer  approves  the  final  screen  design  or  indicates  corrections  or  other  suggestions  for 
improvement.  After  these  revisions,  the  lesson  is  ready  to  be  implemented  in  the  training 
environment. 

4.0  Implement  Training  Program.  While  multimedia  can  have  some  effect  on  the 
implementation  phase  of  a  training  program,  the  basic  steps  remain  unchanged.  For  an 
explanation  of  the  implementation  phase  of  ISD  see  AFP  50-58,  Volume  V. 

5.0  Evaluate  Process  and  Products.  The  steps  performed  in  training  evaluation  can  be  found  in 
AFP  50-58,  Volumes  IV  and  V.  Multimedia  imposes  additional  requirements  on  the  evaluation 
process  since  it  makes  use  of  new  visualization  technology  to  deliver  instruction  and  assess 
student  performance,  and  interjects  different  development  steps  into  the  ISD  model. 
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Data  Dictionary 


Air  Force  commands: 

Animation  requirements: 

Animation: 

Approved  lesson: 
Approved  script: 

Audio  data: 

Audio  element: 

Audio  files: 

Audio  level: 

Audio  log: 

Audio  narration: 

Audio  script: 


Users  of  the  products  of  training  organizations.  The  commands 
received  trained  personnel,  graduates  of  courses,  to  perform  their 
mission(s).  After  a  period  on  the  job,  commands  provide  field 
evaluation  data  on  the  graduates  they  receive. 

Elements  of  the  storyboard  which  can  be  achieved  by  animation 
sequences  rather  than  by  other  means  such  as  simple  graphics,  still 
image  video,  or  full  motion  video. 

Placing  several  graphic  segments  together  to  a  produce  motion 
sequence. 

Lesson  has  been  accepted  for  approval  by  the  customer;  lesson  is 
ready  to  be  implemented. 

Once  the  script  has  been  reviewed  it  is  either  sent  back  for 
revisions  or  approved  for  use  as  is. 

Information  regarding  the  recording  of  audio  either  narration, 
music  or  sound  effects. 

Elements  of  the  storyboard  which  can  be  achieved  by  audio 
sequences  rather  than  by  other  means  such  as  text,  sound  effects, 
music,  video  or  graphics.. 

Storage  location  for  digitized  audio. 

Specification  of  standard  for  audio  level. 

Database  of  information  regarding  audio  and  its  recording 
activities.  Includes  such  audio  data  items  as  lesson  or  storyboard 
to  which  segment(s)  or  entire  tape  relates,  recording  segment  size, 
i.e.,  running  time,  what  is  recorded  on  a  tape,  location  of  segment 
on  the  tape,  and  other  such  pertinent  information. 

Narrative  produced  by  the  talent  in  accordance  with  the  audio 
script.  It  is  normally  recorded  onto  tape  for  later  digitization  and 
overlay  onto  video  segments. 

Document  which  controls  the  production  of  audio.  Lists  the 
storyboard  from  which  a  segment  is  derived,  and  the  actual 
narration,  sound  effects  or  music  required. 


Audio: 


Sound  effects,  music  or  narration  sequences  used  in  producing  a 
multimedia  lesson. 


Color:  Specification  of  standard  for  color. 

Commercial  vendor:  Organization  which  provides  a  product  or  service  needed  during 

the  production  of  multimedia  elements. 

Copyright  OK:  Indication  that  the  sound  segment  being  used  has  no  copyright  or 

that  a  copyright  release  has  been  obtained. 

Customer  comments:  Customer  provides  list  of  comments  regarding  final  screen  design 

during  testing  and  evaluation;  after  revision,  lesson  is  ready  for 
implementation. 

Digitization  schedule:  A  schedule  for  digitizing  (e.g.,  converting  an  analog  signal  into  a 

digital  signal)  multimedia  elements,  listing  the  steps  which  must  be 
done  and  how  long  they  will  take,  is  produced  from  the  multimedia 
elements  list. 

Digitization  sheet:  The  digitization  sheet  is  produced  from  the  multimedia  elements 

list;  for  each  multimedia  element  the  list  contains  the  file  name, 
file  number,  file  size,  and  file  location,  and  information  about 
quality. 

Digitized  audio:  Narration,  sound  effects  or  music  which  has  been  converted  to 

digital  format. 

Digitized  video:  Video  which  has  been  captured  by  a  digitizing  or  capture  routine 

for  use  as  a  digital  file  rather  than  in  traditional  analog  format. 

Edit  list:  This  list,  produced  after  files  are  digitized,  describes  the  placement, 

sequence,  and  rationale  for  editing  each  multimedia  element. 

Edited  files:  After  video  and  audio  have  been  digitized  they  can  be  edited  in 

various  ways  by  splicing  segments  together,  overlaying  audio 
tracks  on  video  segments,  re-ordering  segments,  etc. 

Edited  shot  sheet:  Original  shot  sheet  is  edited  and  revised  to  ensure  that  directions 

for  shooting  new  video  segments  are  clear  and  complete. 

Equipment  ready:  Indication  that  all  equipment  needed  for  a  recording  session  is 

ready  to  operate  as  specified  in  the  audio  or  video  script. 
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Equipment  requirements:  Either  audio  or  video  equipment  required  for  a  recording  session. 

This  may  also  include  other  equipment  such  &c  machines  or  tools 
from  the  job  which  are  to  be  used  in  the  segments  being  recorded 
during  the  session. 

Existing  video:  Video  resources  are  provided  by  the  customer,  obtained  from  a  clip 

library,  or  other  source;  if  quality  is  acceptable,  may  reduce  need 
for  shooting  new  video. 

Feedback:  Data  uncovered  by  the  evaluation  process  which  can  be  of  use  in 

tailoring  or  modifying  the  various  other  activities  of  ISD. 

Field  evaluation  data:  Data  from  various  Air  Force  commands  which  indicates  the  quality 

of  the  graduates  of  training  programs  provided  to  them.  Schools 
can  infer  as  to  the  effectiveness  and  efficiency  of  their  programs 
based  on  this  feedback. 

Video  segment  which  depicts  some  activity  or  function  using 
motion  as  a  component  of  the  scene.  In  contrast  to  still  image 
video  or  graphic  without  animation. 

Students  who  have  completed  some  course  of  instruction 
successfully.  They  normally  report  to  some  major  Air  Force 
command  upon  completion  of  the  training  course. 

Storage  location  for  graphics. 

Contains  graphic  requirements,  video  image  requirements  and 
animation  requirements.  The  graphic  list  is  used  in  developing 
each  of  these  elements. 

Graphic  OK  Indication  that  the  graphic  being  used  has  been  checked  for 

conformance  with  project  standards. 

Graphic  requirements:  Specific  graphics  required  by  a  storyboard.  All  graphic 

requirements  are  entered  into  the  graphic  list. 

Graphic:  A  picture  or  other  depiction  created  with  standardized  software 

packages.  All  graphics  are  developed  from  storyboard 
requirements  and  stored  in  graphic  files. 

Hardware/software:  Computer  hardware  and  software  (coiifiguration)  requirements  for 

the  training  system  are  often  specified  by  the  customer;  for 
example,  working  within  a  particular  operating  environment,  or 
using  a  special  authoring  tool. 


Full  motion  video: 

Graduates: 

Graphic  files: 
Graphic  list: 
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Inflection: 

Specific  areas  of  narration  in  the  audio  script  where  the  talent  is 
directed  to  provide  some  voice  inflection  or  other  characteristic  to 
help  achieve  the  learning  outcomes  specified  in  the  objectives. 

Instructional  design: 

Structural  and  functional  framework  for  presenting  and  evaluating 
instructional  objectives  in  the  training  system. 

Interactive  courseware: 

Computer-based  training  lesson  materials.  It  may  consist  of 
digitized  video  clips,  interactive  videodisc  segments,  CD-ROM 
portions,  graphics,  some  type  of  student  administration  module, 
and  the  software  and  programs  necessary  to  run  the  lessons. 

Job/task  data: 

Job  and  task  analysis  information,  tech  orders  and  manuals,  job 
books,  and  other  materials  which  tell  how  to  perform  the  tasks 
associated  with  a  particular  job.  Customer  provides  documents 
about  subject  matter,  current  training,  student  characteristics, 
photographs,  video,  and  other  relevant  materials  for  analysis  of 
user  and  training  requirements. 

Lesson  prototype: 

Developed  early  in  the  project,  this  is  an  original  model  of  a  frame 
or  lesson  from  which  subsequent  frames  or  lessons  are  patterned. 

Lesson: 

A  complete  interactive  courseware  component  which  is  normally 
accomplished  by  a  student  in  one  sitting.  It  may  include 
multimedia  elements. 

Male-Female: 

Type  of  voice  talent  to  be  in  a  narration. 

Milestone  Chart: 

Schedule  which  lists  major  phases  and  goals  along  with  their 
estimated  time  allotted  for  completion. 

Multimedia  elements  list: 

Using  the  storyboards  and  script/narration,  multimedia  elements 
are  categorized  according  to  type  (i.e.,  graphic,  still  image,  etc.) 
and  specifies  their  file  name,  file  number,  size,  location,  and  other 
features. 

Music: 

Pre-recorded  music  which  can  be  used  in  an  application  instead  of 
producing  original  music. 

Needs: 

Audio,  video,  equipment,  talent  or  other  items  needed  for  a 
recording  session  which  are  not  available.  These  items  must  be 
obtained  from  a  commercial  vendor. 
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New  video  footage: 

Notes: 

Objectives  &  Tests: 

Order: 

Product: 

Production  schedule: 

Programmed  lesson: 

Raw  video: 
Re-record: 

Ready: 

Revisions: 

Schedule  impact: 
Script/narration: 


After  reviewing  existing  video  resources,  the  multimedia  developer 
specifies  new  video  footage  which  is  required  for  producing  the 
lesson. 

Video  data  and  other  information  about  a  recording  session  which 
are  recorded  in  the  video  log. 

Criterion  objectives  and  the  test  instruments  which  measure 
objective  accomplishment  (see  AFP  50-58,  Volume  HI). 

Purchase  order  to  a  commercial  vendor  for  some  item  needed 
during  a  recording  session. 

Item(s)  provided  by  a  commercial  vendor  per  purchase  order. 

Schedule  for  the  production  of  video  and  audio  for  an  application. 
The  schedule  contains  the  items  to  be  produced,  when  the 
production  takes  place,  how  long  production  activities  last, 
resource  requirements,  and  any  other  pertinent  information.  This 
plan,  includes  estimates  of  cost,  time,  materials,  and  personnel 
required  to  complete  each  production  process. 

All  components  of  lesson  have  been  programmed;  lesson  is  ready 
for  testing. 

Video  which  has  just  been  recorded;  no  editing  has  taken  place. 

If  a  segment  is  not  correct  or  does  not  meet  standards  it  must  be  re¬ 
recorded. 

Indication  that  the  equipment,  set.  Camera,  lighting,  format  and 
other  aspects  of  a  recording  session  are  in  place  and  prepared  for 
the  shoot. 

Changes  or  modifications  required  in  interactive  courseware  based 
on  the  results  of  courseware  validation  activities. 

What  effect  transportation  requirements  may  have  on  the 
production  schedule. 

A  document  which  contains  the  segments  recorded  by  the  narrator 
in  addition  to  specific  directions  on  how  segments  are  to  be  read 
and  what  other  multimedia  elements  accompany  each  segment. 
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Shot  sheet: 

SME's  comments: 


Sound  effects  data: 
Sound  effects  script: 

Sound  effects: 

Standards: 

Still  image: 

Storyboards: 


Studio  or  location: 

Talent  ready: 
Talent: 

Training  course: 


Describes  the  number,  type  and  location  of  the  various  video  shots 
which  must  be  recorded  during  production. 

The  subject  matter  expert  (SME)  provides  comments  on  the 
application's  content  and  interactive  design,  graphic  and  production 
design,  hardware  and  software  requirements,  and  instructional 
design. 

Information  regarding  the  recording  of  sound  effects  or  music. 

Script  which  specifies  the  music  or  sound  effects  necessary  for  the 
storyboard(s). 

Specific  areas  of  the  audio  script  where  special  sounds  or  music 
must  be  provided  to  help  achieve  the  learning  outcomes  specified 
in  the  objectives. 

Specifications  for  the  format,  appearance,  files  and  other  aspects  of 
the  lessons.  These  specify  the  minimum  acceptable  standard  for 
the  item. 

Video  produced  specifically  for  showing  details  without  motion, 
i.e.,  single  frame  of  video  is  used  for  each  function  or  element 
depicted. 

Produced  during  the  development  phase  of  ISD  to  describe  the 
content  and  approach  taken  during  the  development  of  interactive 
courseware  lessons.  Consists  of  sequences  of  rough  sketches  that 
show  key  frames  of  the  structure  and  content  of  the  lesson  at 
different  stages. 

Location  where  recording  of  video  sequence  takes  place;  either  on 
location  at  some  job  site  or  in  a  studio  where  the  environment  can 
be  controlled. 

Talent  has  been  coached  and  is  ready  to  record  the  segment. 

Voice,  sound  effects  or  music  provider;  may  be  from  the  same 
organization  or  from  some  commercial  vendor. 

The  product  of  training  development  activities.  Consists  of 
curriculum  materials,  hardware  and  software,  interactive 
courseware,  student  materials,  instructor  materials,  and  other 
elements  used  to  train. 
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Training  requirements:  Product  of  the  analysis  phase  of  ISD  (see  AFP  50-58,  Volume  II). 

T ransportation  requirements:  Need  to  transport  personnel  and/or  equipment  to  a  location  or 

studio  for  a  recording  session. 

Video  data:  Information  regarding  the  recording  of  video  either  on  tape  or 

directly  into  a  digitized  format. 

Video  files:  Storage  location  for  digitized  video. 

Video  format:  Specification  of  standard  for  video  format. 

Video  image  requirements:  Specific  video  images  required  by  a  stoiyboard  for  development 

into  graphics.  All  still  image  requirements  are  entered  into  the 
graphic  list. 

Video  log:  Database  of  information  regarding  video  and  its  recording 

activities.  Includes  such  video  data  items  as  file  names,  file  size, 
scenes  recorded  on  a  tape,  location  of  scenes,  and  other  such 
pertinent  information. 

Video:  Still  images  and  full  motion  video  used  in  producing  a  multimedia 

lesson. 
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